[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:57:58 dist-test-slave-7f01 krb5kdc[22321](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:57:58 dist-test-slave-7f01 krb5kdc[22321](info): set up 2 sockets
Aug 12 00:57:58 dist-test-slave-7f01 krb5kdc[22321](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260812 00:58:00.331679 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.462s user 0.003s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 00:58:00 dist-test-slave-7f01 krb5kdc[22321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:58:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:58:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 00:58:00.574400 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:38129
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:38129
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:01.018105 22340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:01.018503 22340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:01.018697 22340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:01.027024 22340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:58:01.027187 22340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:01.027240 22340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:01.027355 22340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:58:01.027439 22340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:58:01.038762 22340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:38129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:38129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:01.040565 22340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:01.042726 22340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:01.061563 22349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:01.062212 22351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:01.062776 22355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:01.063423 22340 server_base.cc:1034] running on GCE node
I20260812 00:58:01.064870 22340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:01.067139 22340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:01.070611 22340 hybrid_clock.cc:648] HybridClock initialized: now 1786496281070500 us; error 171 us; skew 500 ppm
I20260812 00:58:01.071245 22340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:01 dist-test-slave-7f01 krb5kdc[22321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496281, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:01.080868 22340 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 00:58:01.083418 22340 webserver.cc:460] Webserver started at http://127.21.201.126:45465/ using document root <none> and password file <none>
I20260812 00:58:01.084652 22340 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:01.084858 22340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:01.085282 22340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:01.090152 22340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "3f4c54a144e447e7bb999ab93c771f63"
format_stamp: "Formatted at 2026-08-12 00:58:01 on dist-test-slave-7f01"
server_key: "4a0bf57db96d1eb96b393875a0b7f1e5"
server_key_iv: "7cf6befcef216b5432ed6d116d1bb854"
server_key_version: "encryptionkey@0"
I20260812 00:58:01.091871 22340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "3f4c54a144e447e7bb999ab93c771f63"
format_stamp: "Formatted at 2026-08-12 00:58:01 on dist-test-slave-7f01"
server_key: "4a0bf57db96d1eb96b393875a0b7f1e5"
server_key_iv: "7cf6befcef216b5432ed6d116d1bb854"
server_key_version: "encryptionkey@0"
I20260812 00:58:01.099299 22340 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260812 00:58:01.105224 22367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:01.107551 22340 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260812 00:58:01.108083 22340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "3f4c54a144e447e7bb999ab93c771f63"
format_stamp: "Formatted at 2026-08-12 00:58:01 on dist-test-slave-7f01"
server_key: "4a0bf57db96d1eb96b393875a0b7f1e5"
server_key_iv: "7cf6befcef216b5432ed6d116d1bb854"
server_key_version: "encryptionkey@0"
I20260812 00:58:01.108395 22340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:01.131260 22340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:01.136736 22340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:01.137089 22340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:01.164520 22340 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:38129
I20260812 00:58:01.164709 22443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:38129 every 8 connection(s)
I20260812 00:58:01.166278 22340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 00:58:01.171047 22448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:01.174224 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 22340
I20260812 00:58:01.174628 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 00:58:01.177515 22309 external_mini_cluster.cc:1428] Setting key 6021df57934734934113125f8a9ddbcf
I20260812 00:58:01.187410 22448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f4c54a144e447e7bb999ab93c771f63: Bootstrap starting.
I20260812 00:58:01.191804 22448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f4c54a144e447e7bb999ab93c771f63: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:01.193404 22448 log.cc:824] T 00000000000000000000000000000000 P 3f4c54a144e447e7bb999ab93c771f63: Log is configured to *not* fsync() on all Append() calls
I20260812 00:58:01.198181 22448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f4c54a144e447e7bb999ab93c771f63: No bootstrap required, opened a new log
I20260812 00:58:01.206869 22448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f4c54a144e447e7bb999ab93c771f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4c54a144e447e7bb999ab93c771f63" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38129 } }
I20260812 00:58:01.207486 22448 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f4c54a144e447e7bb999ab93c771f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:01.207659 22448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f4c54a144e447e7bb999ab93c771f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f4c54a144e447e7bb999ab93c771f63, State: Initialized, Role: FOLLOWER
I20260812 00:58:01.208551 22448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f4c54a144e447e7bb999ab93c771f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4c54a144e447e7bb999ab93c771f63" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38129 } }
I20260812 00:58:01.208858 22448 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f4c54a144e447e7bb999ab93c771f63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:58:01.209017 22448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f4c54a144e447e7bb999ab93c771f63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:58:01.209203 22448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f4c54a144e447e7bb999ab93c771f63 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:01.211714 22448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f4c54a144e447e7bb999ab93c771f63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4c54a144e447e7bb999ab93c771f63" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38129 } }
I20260812 00:58:01.212409 22448 leader_election.cc:302] T 00000000000000000000000000000000 P 3f4c54a144e447e7bb999ab93c771f63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f4c54a144e447e7bb999ab93c771f63; no voters:
I20260812 00:58:01.213178 22448 leader_election.cc:288] T 00000000000000000000000000000000 P 3f4c54a144e447e7bb999ab93c771f63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:58:01.213335 22459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f4c54a144e447e7bb999ab93c771f63 [term 1 FOLLOWER]: Leader election won for term 1
Aug 12 00:58:01 dist-test-slave-7f01 krb5kdc[22321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:01.215149 22459 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f4c54a144e447e7bb999ab93c771f63 [term 1 LEADER]: Becoming Leader. State: Replica: 3f4c54a144e447e7bb999ab93c771f63, State: Running, Role: LEADER
I20260812 00:58:01.216192 22448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f4c54a144e447e7bb999ab93c771f63 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:58:01.216037 22459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f4c54a144e447e7bb999ab93c771f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4c54a144e447e7bb999ab93c771f63" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38129 } }
I20260812 00:58:01.226178 22466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f4c54a144e447e7bb999ab93c771f63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f4c54a144e447e7bb999ab93c771f63. Latest consensus state: current_term: 1 leader_uuid: "3f4c54a144e447e7bb999ab93c771f63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4c54a144e447e7bb999ab93c771f63" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38129 } } }
I20260812 00:58:01.226619 22466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f4c54a144e447e7bb999ab93c771f63 [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:01.225689 22464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f4c54a144e447e7bb999ab93c771f63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f4c54a144e447e7bb999ab93c771f63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4c54a144e447e7bb999ab93c771f63" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38129 } } }
I20260812 00:58:01.227001 22464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f4c54a144e447e7bb999ab93c771f63 [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:01.236616 22471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:58:01.239223 22451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:01.183158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43710 (local address 127.21.201.126:38129)
0812 00:58:01.184466 (+ 1308us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:01.184490 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:01.186388 (+ 1898us) server_negotiation.cc:407] Connection header received
0812 00:58:01.198105 (+ 11717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:01.198140 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:01.198717 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:01.199241 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:01.202805 (+ 3564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:01.204647 (+ 1842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:01.206749 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:01.207825 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:01.208068 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:01.215678 (+ 7610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:01.215692 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:01.215775 (+ 83us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:01.215851 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:01.226383 (+ 10532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:01.228677 (+ 2294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:01.228692 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:01.228705 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:01.229225 (+ 520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:01.230170 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:01.230179 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:01.230185 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:01.231506 (+ 1321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:01.236039 (+ 4533us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:01.236503 (+ 464us) server_negotiation.cc:299] Negotiation successful
0812 00:58:01.237315 (+ 812us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":992,"thread_start_us":689,"threads_started":1}
I20260812 00:58:01.249635 22471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:58:01.263087 22471 catalog_manager.cc:1347] Generated new cluster ID: b0fbec6fb89e40e082cc8dd297575fae
I20260812 00:58:01.263280 22471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:58:01.282182 22471 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:58:01.284330 22471 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:58:01.300693 22471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f4c54a144e447e7bb999ab93c771f63: Generated new TSK 0
I20260812 00:58:01.302075 22471 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.201.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
I20260812 00:58:01.384752 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:0
--local_ip_for_outbound_sockets=127.21.201.65
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38129
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:01.842229 22497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:01.842806 22497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:01.843053 22497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:01.853595 22497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:01.853844 22497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:01.854071 22497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 00:58:01.868237 22497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38129
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:01.869976 22497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:01.871654 22497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:01.893726 22510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:01.893874 22511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:01.893925 22513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:01.894966 22497 server_base.cc:1034] running on GCE node
I20260812 00:58:01.895908 22497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:01.897214 22497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:01.898705 22497 hybrid_clock.cc:648] HybridClock initialized: now 1786496281898493 us; error 65 us; skew 500 ppm
I20260812 00:58:01.899302 22497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:01 dist-test-slave-7f01 krb5kdc[22321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496281, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:01.904474 22497 init.cc:377] Logged in from keytab as kudu/127.21.201.65@KRBTEST.COM (short username kudu)
I20260812 00:58:01.906936 22497 webserver.cc:460] Webserver started at http://127.21.201.65:40893/ using document root <none> and password file <none>
I20260812 00:58:01.907874 22497 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:01.908059 22497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:01.908453 22497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:01.912252 22497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-0/data/instance:
uuid: "82e2bfe40565481097bc4f42771ce555"
format_stamp: "Formatted at 2026-08-12 00:58:01 on dist-test-slave-7f01"
server_key: "f3b724ee888fd287ccda1b9ded7eb61a"
server_key_iv: "d790488bf683d914b06011abd00b363f"
server_key_version: "encryptionkey@0"
I20260812 00:58:01.913291 22497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance:
uuid: "82e2bfe40565481097bc4f42771ce555"
format_stamp: "Formatted at 2026-08-12 00:58:01 on dist-test-slave-7f01"
server_key: "f3b724ee888fd287ccda1b9ded7eb61a"
server_key_iv: "d790488bf683d914b06011abd00b363f"
server_key_version: "encryptionkey@0"
I20260812 00:58:01.921291 22497 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260812 00:58:01.926558 22522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:01.928687 22497 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 00:58:01.929060 22497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "82e2bfe40565481097bc4f42771ce555"
format_stamp: "Formatted at 2026-08-12 00:58:01 on dist-test-slave-7f01"
server_key: "f3b724ee888fd287ccda1b9ded7eb61a"
server_key_iv: "d790488bf683d914b06011abd00b363f"
server_key_version: "encryptionkey@0"
I20260812 00:58:01.929384 22497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:01.954295 22497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:01.960206 22497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:01.960677 22497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:01.961903 22497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:58:01.963696 22497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:01.963855 22497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 00:58:01.964033 22497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:01.964115 22497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:02.015069 22497 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:36857
I20260812 00:58:02.015118 22698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:36857 every 8 connection(s)
I20260812 00:58:02.016723 22497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 00:58:02.019240 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 22497
I20260812 00:58:02.019610 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance
I20260812 00:58:02.020668 22309 external_mini_cluster.cc:1428] Setting key d99d0ec4a2a5f8ade6f031b7c7549c30
Aug 12 00:58:02 dist-test-slave-7f01 krb5kdc[22321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496281, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:02.044359 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:02.021518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:59293 (local address 127.21.201.126:38129)
0812 00:58:02.022487 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:02.022495 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:02.023805 (+ 1310us) server_negotiation.cc:407] Connection header received
0812 00:58:02.026006 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:02.026015 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:02.026141 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:02.026359 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:02.028321 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:02.029811 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:02.031849 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:02.032321 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:02.032511 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:02.037091 (+ 4580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:02.037097 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:02.037101 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:02.037143 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:02.040621 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:02.041607 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:02.041611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:02.041613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:02.041768 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:02.042288 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:02.042292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:02.042294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:02.042622 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:02.042791 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:02.043633 (+ 842us) server_negotiation.cc:299] Negotiation successful
0812 00:58:02.043955 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":827,"thread_start_us":678,"threads_started":1}
WARNING: no policy specified for kudu/127.21.201.66@KRBTEST.COM; defaulting to no policy
I20260812 00:58:02.045570 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:02.022381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38129 (local address 127.21.201.65:59293)
0812 00:58:02.023528 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:02.023601 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:02.025659 (+ 2058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:02.026607 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:02.026616 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:02.027271 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:02.028055 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:02.028080 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:02.030077 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:02.030082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:02.031541 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:02.031558 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:02.032643 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:02.032651 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:02.032948 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:02.033882 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:02.033916 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:02.036747 (+ 2831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:02.040840 (+ 4093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:02.040852 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:02.040885 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:02.041440 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:02.041908 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:02.041913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:02.041917 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:02.042146 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:02.042764 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:02.042774 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:02.043202 (+ 428us) client_negotiation.cc:769] Sending connection context
0812 00:58:02.043519 (+ 317us) client_negotiation.cc:240] Negotiation successful
0812 00:58:02.044197 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"thread_start_us":527,"threads_started":1}
I20260812 00:58:02.048323 22703 heartbeater.cc:347] Connected to a master server at 127.21.201.126:38129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.66@KRBTEST.COM" created.
I20260812 00:58:02.048887 22703 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:02.049983 22703 heartbeater.cc:511] Master 127.21.201.126:38129 requested a full tablet report, sending...
I20260812 00:58:02.054203 22394 ts_manager.cc:194] Registered new tserver with Master: 82e2bfe40565481097bc4f42771ce555 (127.21.201.65:36857)
I20260812 00:58:02.057479 22394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.65@KRBTEST.COM'} at 127.21.201.65:59293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
I20260812 00:58:02.085884 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:0
--local_ip_for_outbound_sockets=127.21.201.66
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38129
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:02.514834 22712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:02.515304 22712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:02.515430 22712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:02.525035 22712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:02.525254 22712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:02.525403 22712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 00:58:02.538569 22712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38129
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:02.540292 22712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:02.541899 22712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:02.558024 22720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:02.558212 22722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:02.558786 22726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:02.559574 22712 server_base.cc:1034] running on GCE node
I20260812 00:58:02.560730 22712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:02.562264 22712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:02.563619 22712 hybrid_clock.cc:648] HybridClock initialized: now 1786496282563529 us; error 81 us; skew 500 ppm
I20260812 00:58:02.564123 22712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:02 dist-test-slave-7f01 krb5kdc[22321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496282, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:02.568506 22712 init.cc:377] Logged in from keytab as kudu/127.21.201.66@KRBTEST.COM (short username kudu)
I20260812 00:58:02.570927 22712 webserver.cc:460] Webserver started at http://127.21.201.66:40403/ using document root <none> and password file <none>
I20260812 00:58:02.572345 22712 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:02.572645 22712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:02.573137 22712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:02.577404 22712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-1/data/instance:
uuid: "b28cdd23c3da4f2982b24d2701b834d3"
format_stamp: "Formatted at 2026-08-12 00:58:02 on dist-test-slave-7f01"
server_key: "7b342e30b41825c6703f0aa7afe3e92f"
server_key_iv: "105164f127ff7a65ba6a37522bef3096"
server_key_version: "encryptionkey@0"
I20260812 00:58:02.578436 22712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance:
uuid: "b28cdd23c3da4f2982b24d2701b834d3"
format_stamp: "Formatted at 2026-08-12 00:58:02 on dist-test-slave-7f01"
server_key: "7b342e30b41825c6703f0aa7afe3e92f"
server_key_iv: "105164f127ff7a65ba6a37522bef3096"
server_key_version: "encryptionkey@0"
I20260812 00:58:02.586199 22712 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260812 00:58:02.592139 22738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:02.593869 22712 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 00:58:02.594121 22712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "b28cdd23c3da4f2982b24d2701b834d3"
format_stamp: "Formatted at 2026-08-12 00:58:02 on dist-test-slave-7f01"
server_key: "7b342e30b41825c6703f0aa7afe3e92f"
server_key_iv: "105164f127ff7a65ba6a37522bef3096"
server_key_version: "encryptionkey@0"
I20260812 00:58:02.594393 22712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:02.618710 22712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:02.624008 22712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:02.624379 22712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:02.625782 22712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:58:02.627681 22712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:02.627830 22712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260812 00:58:02.627997 22712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:02.628075 22712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:02.683486 22712 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:46383
I20260812 00:58:02.683552 22926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:46383 every 8 connection(s)
I20260812 00:58:02.685106 22712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 00:58:02.686537 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 22712
I20260812 00:58:02.686848 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance
I20260812 00:58:02.687592 22309 external_mini_cluster.cc:1428] Setting key 511e041a9e320fec5a15208d85c9c305
Aug 12 00:58:02 dist-test-slave-7f01 krb5kdc[22321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496282, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.201.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.67@KRBTEST.COM" created.
I20260812 00:58:02.716348 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:02.688625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:42405 (local address 127.21.201.126:38129)
0812 00:58:02.689192 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:02.689197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:02.690722 (+ 1525us) server_negotiation.cc:407] Connection header received
0812 00:58:02.694612 (+ 3890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:02.694617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:02.694728 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:02.694843 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:02.697505 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:02.698717 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:02.701685 (+ 2968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:02.702590 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:02.702811 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:02.707868 (+ 5057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:02.707879 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:02.707887 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:02.707951 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:02.712077 (+ 4126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:02.713145 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:02.713151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:02.713153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:02.713324 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:02.714111 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:02.714115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:02.714117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:02.714446 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:02.714576 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:02.715771 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0812 00:58:02.716050 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":326,"threads_started":1}
I20260812 00:58:02.718124 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:02.689410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38129 (local address 127.21.201.66:42405)
0812 00:58:02.690531 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:02.690588 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:02.694134 (+ 3546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:02.695213 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:02.695229 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:02.696252 (+ 1023us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:02.697226 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:02.697259 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:02.699185 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:02.699194 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:02.701391 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:02.701423 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:02.702997 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:02.703005 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:02.703306 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:02.704192 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:02.704219 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:02.707421 (+ 3202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:02.712303 (+ 4882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:02.712313 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:02.712329 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:02.712955 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:02.713458 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:02.713462 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:02.713465 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:02.713893 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:02.714582 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:02.714589 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:02.715229 (+ 640us) client_negotiation.cc:769] Sending connection context
0812 00:58:02.715545 (+ 316us) client_negotiation.cc:240] Negotiation successful
0812 00:58:02.716241 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":482,"threads_started":1}
I20260812 00:58:02.720252 22929 heartbeater.cc:347] Connected to a master server at 127.21.201.126:38129
I20260812 00:58:02.720640 22929 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:02.721513 22929 heartbeater.cc:511] Master 127.21.201.126:38129 requested a full tablet report, sending...
I20260812 00:58:02.724032 22394 ts_manager.cc:194] Registered new tserver with Master: b28cdd23c3da4f2982b24d2701b834d3 (127.21.201.66:46383)
I20260812 00:58:02.725271 22394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.66@KRBTEST.COM'} at 127.21.201.66:42405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
I20260812 00:58:02.753103 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:0
--local_ip_for_outbound_sockets=127.21.201.67
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38129
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:58:03.061793 22703 heartbeater.cc:503] Master 127.21.201.126:38129 was elected leader, sending a full tablet report...
W20260812 00:58:03.182849 22935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:03.183233 22935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:03.183413 22935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:03.193382 22935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:03.193544 22935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:03.193758 22935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 00:58:03.205664 22935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38129
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:03.207283 22935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:03.208839 22935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:03.227263 22944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:03.227187 22947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:03.227275 22943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:03.229223 22935 server_base.cc:1034] running on GCE node
I20260812 00:58:03.230384 22935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:03.232107 22935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:03.233534 22935 hybrid_clock.cc:648] HybridClock initialized: now 1786496283233344 us; error 83 us; skew 500 ppm
I20260812 00:58:03.234134 22935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:03 dist-test-slave-7f01 krb5kdc[22321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496283, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:03.238597 22935 init.cc:377] Logged in from keytab as kudu/127.21.201.67@KRBTEST.COM (short username kudu)
I20260812 00:58:03.241009 22935 webserver.cc:460] Webserver started at http://127.21.201.67:45663/ using document root <none> and password file <none>
I20260812 00:58:03.241859 22935 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:03.241983 22935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:03.242270 22935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:03.245884 22935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-2/data/instance:
uuid: "e0f27dad007f462b84053310df348a01"
format_stamp: "Formatted at 2026-08-12 00:58:03 on dist-test-slave-7f01"
server_key: "c3f0dbfa78194187483544106aca7951"
server_key_iv: "deca15f94a439ad94059afe3ef6b4d93"
server_key_version: "encryptionkey@0"
I20260812 00:58:03.246869 22935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance:
uuid: "e0f27dad007f462b84053310df348a01"
format_stamp: "Formatted at 2026-08-12 00:58:03 on dist-test-slave-7f01"
server_key: "c3f0dbfa78194187483544106aca7951"
server_key_iv: "deca15f94a439ad94059afe3ef6b4d93"
server_key_version: "encryptionkey@0"
I20260812 00:58:03.254215 22935 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260812 00:58:03.259060 22961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:03.261087 22935 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 00:58:03.261354 22935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "e0f27dad007f462b84053310df348a01"
format_stamp: "Formatted at 2026-08-12 00:58:03 on dist-test-slave-7f01"
server_key: "c3f0dbfa78194187483544106aca7951"
server_key_iv: "deca15f94a439ad94059afe3ef6b4d93"
server_key_version: "encryptionkey@0"
I20260812 00:58:03.261656 22935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:03.288082 22935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:03.293994 22935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:03.294359 22935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:03.295667 22935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:58:03.297361 22935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:03.297453 22935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:03.297634 22935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:03.297690 22935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:03.348220 22935 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:36843
I20260812 00:58:03.348340 23162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:36843 every 8 connection(s)
I20260812 00:58:03.349699 22935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 00:58:03.354058 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 22935
I20260812 00:58:03.354295 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance
I20260812 00:58:03.354828 22309 external_mini_cluster.cc:1428] Setting key e9daf1d052336bad621f6e3a40e0537b
Aug 12 00:58:03 dist-test-slave-7f01 krb5kdc[22321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496283, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:03.380466 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:03.353066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:47657 (local address 127.21.201.126:38129)
0812 00:58:03.353645 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:03.353649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:03.354802 (+ 1153us) server_negotiation.cc:407] Connection header received
0812 00:58:03.357331 (+ 2529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:03.357337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:03.357438 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:03.357579 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:03.360749 (+ 3170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:03.361611 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:03.363876 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:03.364440 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:03.364572 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:03.369947 (+ 5375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:03.369954 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:03.369958 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:03.370003 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:03.374653 (+ 4650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:03.376491 (+ 1838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:03.376496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:03.376499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:03.376668 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:03.377882 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:03.377889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:03.377892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:03.378435 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:03.378622 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:03.379636 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0812 00:58:03.380041 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":342,"threads_started":1}
I20260812 00:58:03.381421 23169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:03.353594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38129 (local address 127.21.201.67:47657)
0812 00:58:03.354594 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:03.354645 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:03.357042 (+ 2397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:03.358235 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:03.358256 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:03.359400 (+ 1144us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:03.360472 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:03.360498 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:03.361862 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:03.361867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:03.363573 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:03.363592 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:03.364861 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:03.364876 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:03.365169 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:03.366238 (+ 1069us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:03.366275 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:03.369617 (+ 3342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:03.375271 (+ 5654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:03.375287 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:03.375592 (+ 305us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:03.376294 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:03.376926 (+ 632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:03.376930 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:03.376933 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:03.377569 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:03.378615 (+ 1046us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:03.378624 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:03.379129 (+ 505us) client_negotiation.cc:769] Sending connection context
0812 00:58:03.379444 (+ 315us) client_negotiation.cc:240] Negotiation successful
0812 00:58:03.380063 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":397,"threads_started":1}
I20260812 00:58:03.383673 23163 heartbeater.cc:347] Connected to a master server at 127.21.201.126:38129
I20260812 00:58:03.384393 23163 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:03.385461 23163 heartbeater.cc:511] Master 127.21.201.126:38129 requested a full tablet report, sending...
I20260812 00:58:03.388140 22390 ts_manager.cc:194] Registered new tserver with Master: e0f27dad007f462b84053310df348a01 (127.21.201.67:36843)
I20260812 00:58:03.389667 22390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.67@KRBTEST.COM'} at 127.21.201.67:47657
I20260812 00:58:03.389746 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 00:58:03 dist-test-slave-7f01 krb5kdc[22321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496283, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 00:58:03 dist-test-slave-7f01 krb5kdc[22321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496283, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.201.65@KRBTEST.COM
I20260812 00:58:03.461853 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:03.442035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34826 (local address 127.21.201.65:36857)
0812 00:58:03.442799 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:03.442808 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:03.442841 (+ 33us) server_negotiation.cc:407] Connection header received
0812 00:58:03.442921 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:03.442930 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:03.443242 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:03.443436 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:03.445399 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:03.446631 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:03.448401 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:03.449127 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:03.449263 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:03.454360 (+ 5097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:03.454380 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:03.454397 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:03.454444 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:03.457890 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:03.458803 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:03.458812 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:03.458814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:03.458977 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:03.459431 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:03.459435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:03.459437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:03.459948 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:03.460302 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:03.461154 (+ 852us) server_negotiation.cc:299] Negotiation successful
0812 00:58:03.461474 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":384,"threads_started":1}
W20260812 00:58:03.463443 22563 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34826
Aug 12 00:58:03 dist-test-slave-7f01 krb5kdc[22321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 00:58:03 dist-test-slave-7f01 krb5kdc[22321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.65@KRBTEST.COM
I20260812 00:58:03.521137 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:03.506416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34842 (local address 127.21.201.65:36857)
0812 00:58:03.506693 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:03.506698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:03.506787 (+ 89us) server_negotiation.cc:407] Connection header received
0812 00:58:03.506967 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:03.506971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:03.507076 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:03.507168 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:03.508861 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:03.509600 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:03.510902 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:03.511358 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:03.511454 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:03.516013 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:03.516018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:03.516022 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:03.516057 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:03.518678 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:03.519320 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:03.519323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:03.519326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:03.519463 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:03.519828 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:03.519832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:03.519834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:03.520105 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:03.520375 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:03.520585 (+ 210us) server_negotiation.cc:299] Negotiation successful
0812 00:58:03.520758 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260812 00:58:03.522873 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 22497
I20260812 00:58:03.536685 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:03.523933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34854 (local address 127.21.201.65:36857)
0812 00:58:03.524146 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:03.524154 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:03.524480 (+ 326us) server_negotiation.cc:407] Connection header received
0812 00:58:03.524579 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:03.524583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:03.524636 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:03.524713 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:03.525844 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:03.526515 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:03.527917 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:03.528348 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:03.528457 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:03.530770 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:03.530774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:03.530777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:03.530808 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:03.533829 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:03.534558 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:03.534562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:03.534564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:03.534775 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:03.535167 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:03.535171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:03.535174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:03.535505 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:03.535706 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:03.536095 (+ 389us) server_negotiation.cc:299] Negotiation successful
0812 00:58:03.536263 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 00:58:03.537614 22690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:03.624984 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 22497
I20260812 00:58:03.652635 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 22712
Aug 12 00:58:03 dist-test-slave-7f01 krb5kdc[22321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.66@KRBTEST.COM
I20260812 00:58:03.669462 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:03.653606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34260 (local address 127.21.201.66:46383)
0812 00:58:03.654250 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:03.654257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:03.654284 (+ 27us) server_negotiation.cc:407] Connection header received
0812 00:58:03.654358 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:03.654365 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:03.654585 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:03.654719 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:03.656059 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:03.657078 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:03.658392 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:03.658958 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:03.659061 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:03.663648 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:03.663660 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:03.663674 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:03.663727 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:03.666698 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:03.667376 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:03.667383 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:03.667385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:03.667519 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:03.668074 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:03.668078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:03.668080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:03.668464 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:03.668714 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:03.668936 (+ 222us) server_negotiation.cc:299] Negotiation successful
0812 00:58:03.669144 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":339,"threads_started":1}
I20260812 00:58:03.670910 22919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:03.760850 22929 heartbeater.cc:503] Master 127.21.201.126:38129 was elected leader, sending a full tablet report...
I20260812 00:58:03.762982 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 22712
I20260812 00:58:03.794770 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 22935
Aug 12 00:58:03 dist-test-slave-7f01 krb5kdc[22321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.67@KRBTEST.COM
I20260812 00:58:03.817036 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:03.796494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56782 (local address 127.21.201.67:36843)
0812 00:58:03.797212 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:03.797220 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:03.797258 (+ 38us) server_negotiation.cc:407] Connection header received
0812 00:58:03.797335 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:03.797342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:03.797591 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:03.797759 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:03.799208 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:03.800560 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:03.802305 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:03.802933 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:03.803081 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:03.809015 (+ 5934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:03.809033 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:03.809050 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:03.809098 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:03.812938 (+ 3840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:03.814016 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:03.814046 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:03.814051 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:03.814449 (+ 398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:03.815010 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:03.815015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:03.815018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:03.815484 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:03.815842 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:03.816328 (+ 486us) server_negotiation.cc:299] Negotiation successful
0812 00:58:03.816563 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":320,"threads_started":1}
I20260812 00:58:03.818631 23152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:03.918138 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 22935
I20260812 00:58:03.955101 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 22340
I20260812 00:58:03.956792 22438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:04.071961 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 22340
2026-08-12T00:58:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5274 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:58:04 dist-test-slave-7f01 krb5kdc[23202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:58:04 dist-test-slave-7f01 krb5kdc[23202](info): set up 2 sockets
Aug 12 00:58:04 dist-test-slave-7f01 krb5kdc[23202](info): commencing operation
krb5kdc: starting...
W20260812 00:58:05.554124 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.424s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 00:58:05 dist-test-slave-7f01 krb5kdc[23202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:58:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:58:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 00:58:05.886646 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:41763
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:40493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:41763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:06.308139 23221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:06.308521 23221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:06.308688 23221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:06.317461 23221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:58:06.317624 23221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:06.317677 23221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:06.317775 23221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:58:06.317874 23221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:58:06.329344 23221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:40493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:41763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:41763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:06.331020 23221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:06.332705 23221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:06.348426 23227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:06.348440 23226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:06.349462 23229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:06.351315 23221 server_base.cc:1034] running on GCE node
I20260812 00:58:06.352499 23221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:06.354261 23221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:06.355530 23221 hybrid_clock.cc:648] HybridClock initialized: now 1786496286355471 us; error 46 us; skew 500 ppm
I20260812 00:58:06.356119 23221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:06 dist-test-slave-7f01 krb5kdc[23202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496286, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:06.366906 23221 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 00:58:06.369648 23221 webserver.cc:460] Webserver started at http://127.21.201.126:46645/ using document root <none> and password file <none>
I20260812 00:58:06.370627 23221 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:06.370767 23221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:06.371181 23221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:06.374830 23221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "c6855c2b238744e8a27d9be46ecebc30"
format_stamp: "Formatted at 2026-08-12 00:58:06 on dist-test-slave-7f01"
server_key: "8e9ce602a13109e6e06b5f1011858116"
server_key_iv: "bea6080d14e4ec343883d8dc59e04922"
server_key_version: "encryptionkey@0"
I20260812 00:58:06.375850 23221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "c6855c2b238744e8a27d9be46ecebc30"
format_stamp: "Formatted at 2026-08-12 00:58:06 on dist-test-slave-7f01"
server_key: "8e9ce602a13109e6e06b5f1011858116"
server_key_iv: "bea6080d14e4ec343883d8dc59e04922"
server_key_version: "encryptionkey@0"
I20260812 00:58:06.383286 23221 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260812 00:58:06.388605 23237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:06.390640 23221 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 00:58:06.390930 23221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "c6855c2b238744e8a27d9be46ecebc30"
format_stamp: "Formatted at 2026-08-12 00:58:06 on dist-test-slave-7f01"
server_key: "8e9ce602a13109e6e06b5f1011858116"
server_key_iv: "bea6080d14e4ec343883d8dc59e04922"
server_key_version: "encryptionkey@0"
I20260812 00:58:06.391178 23221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:06.415361 23221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:06.420734 23221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:06.421108 23221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:06.444779 23221 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:41763
I20260812 00:58:06.444819 23332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:41763 every 8 connection(s)
I20260812 00:58:06.446971 23221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 00:58:06.447959 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 23221
I20260812 00:58:06.448299 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 00:58:06.449301 22309 external_mini_cluster.cc:1428] Setting key a4b6cc288b1b23ccca41753a3bafab3c
I20260812 00:58:06.453775 23333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:06.467070 23333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6855c2b238744e8a27d9be46ecebc30: Bootstrap starting.
Aug 12 00:58:06 dist-test-slave-7f01 krb5kdc[23202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:06.471585 23333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6855c2b238744e8a27d9be46ecebc30: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:06.472973 23333 log.cc:824] T 00000000000000000000000000000000 P c6855c2b238744e8a27d9be46ecebc30: Log is configured to *not* fsync() on all Append() calls
I20260812 00:58:06.476524 23333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6855c2b238744e8a27d9be46ecebc30: No bootstrap required, opened a new log
I20260812 00:58:06.478487 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:06.452293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58930 (local address 127.21.201.126:41763)
0812 00:58:06.453486 (+ 1193us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:06.453506 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:06.453572 (+ 66us) server_negotiation.cc:407] Connection header received
0812 00:58:06.456280 (+ 2708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:06.456314 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:06.456961 (+ 647us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:06.457527 (+ 566us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:06.459386 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:06.460969 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:06.462416 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:06.463247 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:06.463509 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:06.469100 (+ 5591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:06.469114 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:06.469139 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:06.469224 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:06.473447 (+ 4223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:06.474443 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:06.474452 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:06.474460 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:06.474681 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:06.475136 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:06.475144 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:06.475152 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:06.475882 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:06.476269 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:06.476786 (+ 517us) server_negotiation.cc:299] Negotiation successful
0812 00:58:06.477315 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":868,"thread_start_us":498,"threads_started":1}
I20260812 00:58:06.486274 23333 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6855c2b238744e8a27d9be46ecebc30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6855c2b238744e8a27d9be46ecebc30" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41763 } }
I20260812 00:58:06.486852 23333 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6855c2b238744e8a27d9be46ecebc30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:06.486991 23333 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6855c2b238744e8a27d9be46ecebc30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6855c2b238744e8a27d9be46ecebc30, State: Initialized, Role: FOLLOWER
I20260812 00:58:06.487685 23333 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6855c2b238744e8a27d9be46ecebc30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6855c2b238744e8a27d9be46ecebc30" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41763 } }
I20260812 00:58:06.488070 23333 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6855c2b238744e8a27d9be46ecebc30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:58:06.488247 23333 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6855c2b238744e8a27d9be46ecebc30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:58:06.488466 23333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6855c2b238744e8a27d9be46ecebc30 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:06.491346 23333 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6855c2b238744e8a27d9be46ecebc30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6855c2b238744e8a27d9be46ecebc30" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41763 } }
I20260812 00:58:06.492089 23333 leader_election.cc:302] T 00000000000000000000000000000000 P c6855c2b238744e8a27d9be46ecebc30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6855c2b238744e8a27d9be46ecebc30; no voters:
I20260812 00:58:06.492826 23333 leader_election.cc:288] T 00000000000000000000000000000000 P c6855c2b238744e8a27d9be46ecebc30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:58:06.493028 23340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6855c2b238744e8a27d9be46ecebc30 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:58:06.494554 23340 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6855c2b238744e8a27d9be46ecebc30 [term 1 LEADER]: Becoming Leader. State: Replica: c6855c2b238744e8a27d9be46ecebc30, State: Running, Role: LEADER
I20260812 00:58:06.495225 23340 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6855c2b238744e8a27d9be46ecebc30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6855c2b238744e8a27d9be46ecebc30" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41763 } }
I20260812 00:58:06.495695 23333 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6855c2b238744e8a27d9be46ecebc30 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:58:06.498797 23342 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6855c2b238744e8a27d9be46ecebc30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6855c2b238744e8a27d9be46ecebc30. Latest consensus state: current_term: 1 leader_uuid: "c6855c2b238744e8a27d9be46ecebc30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6855c2b238744e8a27d9be46ecebc30" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41763 } } }
I20260812 00:58:06.499306 23342 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6855c2b238744e8a27d9be46ecebc30 [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:06.498771 23341 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6855c2b238744e8a27d9be46ecebc30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6855c2b238744e8a27d9be46ecebc30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6855c2b238744e8a27d9be46ecebc30" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41763 } } }
I20260812 00:58:06.499432 23341 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6855c2b238744e8a27d9be46ecebc30 [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:06.500762 23350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:58:06.508750 23350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:58:06.521652 23350 catalog_manager.cc:1347] Generated new cluster ID: 0f38ff2463e3459ea8d25592e98bd97f
I20260812 00:58:06.521819 23350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:58:06.533607 23350 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:58:06.535121 23350 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:58:06.545742 23350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6855c2b238744e8a27d9be46ecebc30: Generated new TSK 0
I20260812 00:58:06.547058 23350 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.201.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
I20260812 00:58:06.612003 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:0
--local_ip_for_outbound_sockets=127.21.201.65
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:41763
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:40493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:07.031311 23371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:07.031724 23371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:07.031941 23371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:07.040820 23371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:07.040982 23371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:07.041116 23371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 00:58:07.054559 23371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:40493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:41763
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:07.056159 23371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:07.057672 23371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:07.074365 23380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:07.074312 23388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:07.074422 23379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:07.078157 23371 server_base.cc:1034] running on GCE node
I20260812 00:58:07.079057 23371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:07.080436 23371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:07.081848 23371 hybrid_clock.cc:648] HybridClock initialized: now 1786496287081788 us; error 68 us; skew 500 ppm
I20260812 00:58:07.082311 23371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:07 dist-test-slave-7f01 krb5kdc[23202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496287, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:07.086733 23371 init.cc:377] Logged in from keytab as kudu/127.21.201.65@KRBTEST.COM (short username kudu)
I20260812 00:58:07.089583 23371 webserver.cc:460] Webserver started at http://127.21.201.65:43625/ using document root <none> and password file <none>
I20260812 00:58:07.090747 23371 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:07.090963 23371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:07.091377 23371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:07.095091 23371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-0/data/instance:
uuid: "3d256d98a86643909d6d4a1a3ea65ca9"
format_stamp: "Formatted at 2026-08-12 00:58:07 on dist-test-slave-7f01"
server_key: "629351aed641ea8e4a78069ac0cdc9a3"
server_key_iv: "b023250ddce2ad425556e994d64d0350"
server_key_version: "encryptionkey@0"
I20260812 00:58:07.096218 23371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance:
uuid: "3d256d98a86643909d6d4a1a3ea65ca9"
format_stamp: "Formatted at 2026-08-12 00:58:07 on dist-test-slave-7f01"
server_key: "629351aed641ea8e4a78069ac0cdc9a3"
server_key_iv: "b023250ddce2ad425556e994d64d0350"
server_key_version: "encryptionkey@0"
I20260812 00:58:07.103916 23371 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.002s
I20260812 00:58:07.110069 23396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:07.112128 23371 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.004s
I20260812 00:58:07.112386 23371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "3d256d98a86643909d6d4a1a3ea65ca9"
format_stamp: "Formatted at 2026-08-12 00:58:07 on dist-test-slave-7f01"
server_key: "629351aed641ea8e4a78069ac0cdc9a3"
server_key_iv: "b023250ddce2ad425556e994d64d0350"
server_key_version: "encryptionkey@0"
I20260812 00:58:07.112687 23371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:07.134876 23371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:07.140242 23371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:07.140605 23371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:07.141949 23371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:58:07.143901 23371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:07.144016 23371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:07.144202 23371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:07.144282 23371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:07.186942 23371 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:38897
I20260812 00:58:07.187109 23573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:38897 every 8 connection(s)
I20260812 00:58:07.188676 23371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 00:58:07.189649 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 23371
I20260812 00:58:07.189958 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance
I20260812 00:58:07.190802 22309 external_mini_cluster.cc:1428] Setting key 48b97b84fc6bc0a460522cb0eae7e389
Aug 12 00:58:07 dist-test-slave-7f01 krb5kdc[23202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496287, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.201.66@KRBTEST.COM; defaulting to no policy
I20260812 00:58:07.215293 23578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:07.191941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:53629 (local address 127.21.201.126:41763)
0812 00:58:07.192530 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:07.192536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:07.193884 (+ 1348us) server_negotiation.cc:407] Connection header received
0812 00:58:07.195983 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:07.195992 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:07.196112 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:07.196243 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:07.198292 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:07.199585 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:07.201528 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:07.201962 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:07.202136 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:07.207070 (+ 4934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:07.207078 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:07.207082 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:07.207137 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:07.210572 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:07.211722 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:07.211728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:07.211730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:07.211976 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:07.212626 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:07.212630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:07.212633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:07.212988 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:07.213407 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:07.214519 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0812 00:58:07.214835 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":311,"threads_started":1}
I20260812 00:58:07.216583 23579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:07.192727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:41763 (local address 127.21.201.65:53629)
0812 00:58:07.193670 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:07.193722 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:07.195619 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:07.196663 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:07.196672 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:07.197288 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:07.198101 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:07.198121 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:07.199839 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:07.199848 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:07.201340 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:07.201355 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:07.202256 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:07.202265 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:07.202530 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:07.203320 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:07.203351 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:07.206614 (+ 3263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:07.210835 (+ 4221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:07.210855 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:07.210880 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:07.211534 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:07.212132 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:07.212138 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:07.212142 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:07.212414 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:07.213372 (+ 958us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:07.213385 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:07.213991 (+ 606us) client_negotiation.cc:769] Sending connection context
0812 00:58:07.214336 (+ 345us) client_negotiation.cc:240] Negotiation successful
0812 00:58:07.215059 (+ 723us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":382,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.66@KRBTEST.COM" created.
I20260812 00:58:07.218643 23574 heartbeater.cc:347] Connected to a master server at 127.21.201.126:41763
I20260812 00:58:07.219039 23574 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:07.220121 23574 heartbeater.cc:511] Master 127.21.201.126:41763 requested a full tablet report, sending...
I20260812 00:58:07.223488 23266 ts_manager.cc:194] Registered new tserver with Master: 3d256d98a86643909d6d4a1a3ea65ca9 (127.21.201.65:38897)
I20260812 00:58:07.226162 23266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.65@KRBTEST.COM'} at 127.21.201.65:53629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
I20260812 00:58:07.252137 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:0
--local_ip_for_outbound_sockets=127.21.201.66
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:41763
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:40493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:07.652364 23583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:07.652791 23583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:07.652982 23583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:07.660813 23583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:07.661038 23583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:07.661240 23583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 00:58:07.673259 23583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:40493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:41763
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:07.675259 23583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:07.677469 23583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:07.694986 23589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:07.695264 23593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:07.695122 23590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:07.696645 23583 server_base.cc:1034] running on GCE node
I20260812 00:58:07.697546 23583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:07.698709 23583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:07.700038 23583 hybrid_clock.cc:648] HybridClock initialized: now 1786496287699946 us; error 58 us; skew 500 ppm
I20260812 00:58:07.700657 23583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:07 dist-test-slave-7f01 krb5kdc[23202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496287, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:07.705693 23583 init.cc:377] Logged in from keytab as kudu/127.21.201.66@KRBTEST.COM (short username kudu)
I20260812 00:58:07.708410 23583 webserver.cc:460] Webserver started at http://127.21.201.66:33095/ using document root <none> and password file <none>
I20260812 00:58:07.709514 23583 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:07.709986 23583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:07.710551 23583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:07.715062 23583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-1/data/instance:
uuid: "21abb977fe0747188702c8e263f92661"
format_stamp: "Formatted at 2026-08-12 00:58:07 on dist-test-slave-7f01"
server_key: "342933c4feb317142b01c0db4c4a0995"
server_key_iv: "77a7a3561b4e5df6591201e2ebd59d60"
server_key_version: "encryptionkey@0"
I20260812 00:58:07.716446 23583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance:
uuid: "21abb977fe0747188702c8e263f92661"
format_stamp: "Formatted at 2026-08-12 00:58:07 on dist-test-slave-7f01"
server_key: "342933c4feb317142b01c0db4c4a0995"
server_key_iv: "77a7a3561b4e5df6591201e2ebd59d60"
server_key_version: "encryptionkey@0"
I20260812 00:58:07.724543 23583 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260812 00:58:07.729811 23601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:07.731521 23583 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260812 00:58:07.731823 23583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "21abb977fe0747188702c8e263f92661"
format_stamp: "Formatted at 2026-08-12 00:58:07 on dist-test-slave-7f01"
server_key: "342933c4feb317142b01c0db4c4a0995"
server_key_iv: "77a7a3561b4e5df6591201e2ebd59d60"
server_key_version: "encryptionkey@0"
I20260812 00:58:07.732146 23583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:07.759229 23583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:07.764899 23583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:07.765314 23583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:07.766536 23583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:58:07.768332 23583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:07.768455 23583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:07.768618 23583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:07.768695 23583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:07.812201 23583 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:39205
I20260812 00:58:07.812529 23811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:39205 every 8 connection(s)
I20260812 00:58:07.814011 23583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 00:58:07.820278 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 23583
I20260812 00:58:07.820572 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance
I20260812 00:58:07.821213 22309 external_mini_cluster.cc:1428] Setting key 1e0319eed4993d3e012beaf1666023bf
Aug 12 00:58:07 dist-test-slave-7f01 krb5kdc[23202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496287, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:07.841092 23814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:07.817407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:49707 (local address 127.21.201.126:41763)
0812 00:58:07.817984 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:07.817989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:07.818944 (+ 955us) server_negotiation.cc:407] Connection header received
0812 00:58:07.821251 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:07.821257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:07.821361 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:07.821486 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:07.823700 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:07.824555 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:07.826317 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:07.826848 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:07.827008 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:07.831214 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:07.831220 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:07.831223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:07.831268 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:07.835563 (+ 4295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:07.836825 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:07.836830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:07.836833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:07.837009 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:07.837614 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:07.837622 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:07.837628 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:07.838085 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:07.838281 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:07.839868 (+ 1587us) server_negotiation.cc:299] Negotiation successful
0812 00:58:07.840459 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":325,"threads_started":1}
I20260812 00:58:07.842708 23815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:07.817919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:41763 (local address 127.21.201.66:49707)
0812 00:58:07.818755 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:07.818802 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:07.820927 (+ 2125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:07.821755 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:07.821765 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:07.822558 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:07.823511 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:07.823535 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:07.824743 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:07.824749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:07.826131 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:07.826146 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:07.827133 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:07.827141 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:07.827413 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:07.828304 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:07.828331 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:07.830889 (+ 2558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:07.835809 (+ 4920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:07.835824 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:07.835849 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:07.836669 (+ 820us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:07.837138 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:07.837146 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:07.837153 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:07.837461 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:07.838504 (+ 1043us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:07.838511 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:07.838934 (+ 423us) client_negotiation.cc:769] Sending connection context
0812 00:58:07.839575 (+ 641us) client_negotiation.cc:240] Negotiation successful
0812 00:58:07.840649 (+ 1074us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":329,"threads_started":1}
I20260812 00:58:07.845552 23812 heartbeater.cc:347] Connected to a master server at 127.21.201.126:41763
I20260812 00:58:07.846045 23812 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:07.847146 23812 heartbeater.cc:511] Master 127.21.201.126:41763 requested a full tablet report, sending...
I20260812 00:58:07.849681 23266 ts_manager.cc:194] Registered new tserver with Master: 21abb977fe0747188702c8e263f92661 (127.21.201.66:39205)
WARNING: no policy specified for kudu/127.21.201.67@KRBTEST.COM; defaulting to no policy
I20260812 00:58:07.850755 23266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.66@KRBTEST.COM'} at 127.21.201.66:49707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
I20260812 00:58:07.888957 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:0
--local_ip_for_outbound_sockets=127.21.201.67
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:41763
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:40493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:58:08.230950 23574 heartbeater.cc:503] Master 127.21.201.126:41763 was elected leader, sending a full tablet report...
W20260812 00:58:08.299978 23818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:08.300335 23818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:08.300455 23818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:08.308745 23818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:08.308907 23818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:08.309072 23818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 00:58:08.320597 23818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:40493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:41763
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:08.322065 23818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:08.323546 23818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:08.339104 23828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:08.339149 23830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:08.339071 23827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:08.340122 23818 server_base.cc:1034] running on GCE node
I20260812 00:58:08.340838 23818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:08.342849 23818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:08.344432 23818 hybrid_clock.cc:648] HybridClock initialized: now 1786496288344341 us; error 97 us; skew 500 ppm
I20260812 00:58:08.344983 23818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:08 dist-test-slave-7f01 krb5kdc[23202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496288, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:08.350172 23818 init.cc:377] Logged in from keytab as kudu/127.21.201.67@KRBTEST.COM (short username kudu)
I20260812 00:58:08.352598 23818 webserver.cc:460] Webserver started at http://127.21.201.67:45683/ using document root <none> and password file <none>
I20260812 00:58:08.353401 23818 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:08.353538 23818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:08.353843 23818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:08.357406 23818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-2/data/instance:
uuid: "8f57d307e1bf478681428ce4a7333aa1"
format_stamp: "Formatted at 2026-08-12 00:58:08 on dist-test-slave-7f01"
server_key: "02f48da4f3a3315bb21829b889359671"
server_key_iv: "21e9f3a477b4fe1b5157c87debd2e22e"
server_key_version: "encryptionkey@0"
I20260812 00:58:08.358383 23818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance:
uuid: "8f57d307e1bf478681428ce4a7333aa1"
format_stamp: "Formatted at 2026-08-12 00:58:08 on dist-test-slave-7f01"
server_key: "02f48da4f3a3315bb21829b889359671"
server_key_iv: "21e9f3a477b4fe1b5157c87debd2e22e"
server_key_version: "encryptionkey@0"
I20260812 00:58:08.365219 23818 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 00:58:08.370147 23840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:08.372437 23818 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260812 00:58:08.372717 23818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "8f57d307e1bf478681428ce4a7333aa1"
format_stamp: "Formatted at 2026-08-12 00:58:08 on dist-test-slave-7f01"
server_key: "02f48da4f3a3315bb21829b889359671"
server_key_iv: "21e9f3a477b4fe1b5157c87debd2e22e"
server_key_version: "encryptionkey@0"
I20260812 00:58:08.372960 23818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:08.406033 23818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:08.411412 23818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:08.411712 23818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:08.412946 23818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:58:08.414788 23818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:08.414876 23818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:08.415025 23818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:08.415096 23818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:08.462476 23818 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:34297
I20260812 00:58:08.462636 24019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:34297 every 8 connection(s)
I20260812 00:58:08.464201 23818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 00:58:08.467800 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 23818
I20260812 00:58:08.468173 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance
I20260812 00:58:08.468843 22309 external_mini_cluster.cc:1428] Setting key 28dea78ed9891b7198320392a31fbc5b
Aug 12 00:58:08 dist-test-slave-7f01 krb5kdc[23202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496288, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:08.493250 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:08.468380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:37545 (local address 127.21.201.126:41763)
0812 00:58:08.469001 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:08.469010 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:08.470956 (+ 1946us) server_negotiation.cc:407] Connection header received
0812 00:58:08.473792 (+ 2836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:08.473798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:08.473901 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:08.474032 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:08.475822 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.476746 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.478531 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.478979 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.479122 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:08.484449 (+ 5327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:08.484456 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:08.484460 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:08.484560 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:08.489235 (+ 4675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:08.490475 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:08.490483 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:08.490489 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:08.490647 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:08.491135 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:08.491141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:08.491144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:08.491452 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:08.491630 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:08.492535 (+ 905us) server_negotiation.cc:299] Negotiation successful
0812 00:58:08.492815 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":312,"threads_started":1}
I20260812 00:58:08.494415 24025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:08.469545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:41763 (local address 127.21.201.67:37545)
0812 00:58:08.470771 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:08.470848 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:08.473500 (+ 2652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:08.474314 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:08.474323 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:08.474935 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:08.475607 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:08.475627 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.476934 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.476939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:08.478350 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:08.478364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.479257 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.479265 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:08.479530 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:08.480501 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:08.480538 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:08.484023 (+ 3485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:08.489529 (+ 5506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:08.489545 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:08.489570 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:08.490315 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:08.490774 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:08.490779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:08.490784 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:08.491011 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:08.491593 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:08.491602 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:08.492081 (+ 479us) client_negotiation.cc:769] Sending connection context
0812 00:58:08.492442 (+ 361us) client_negotiation.cc:240] Negotiation successful
0812 00:58:08.493074 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"thread_start_us":586,"threads_started":1}
I20260812 00:58:08.496325 24021 heartbeater.cc:347] Connected to a master server at 127.21.201.126:41763
I20260812 00:58:08.496739 24021 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:08.497617 24021 heartbeater.cc:511] Master 127.21.201.126:41763 requested a full tablet report, sending...
I20260812 00:58:08.499514 23266 ts_manager.cc:194] Registered new tserver with Master: 8f57d307e1bf478681428ce4a7333aa1 (127.21.201.67:34297)
I20260812 00:58:08.500567 23266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.67@KRBTEST.COM'} at 127.21.201.67:37545
I20260812 00:58:08.501171 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:58:08.536295 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:08.511051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35006 (local address 127.21.201.126:41763)
0812 00:58:08.511373 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:08.511384 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:08.511995 (+ 611us) server_negotiation.cc:407] Connection header received
0812 00:58:08.512243 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:08.512249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:08.512357 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:08.512460 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:08.514421 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.515254 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.516802 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.517369 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.517485 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:08.520510 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:08.520523 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:08.520532 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:08.520599 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:08.530952 (+ 10353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:08.533504 (+ 2552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:08.533511 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:08.533515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:08.533727 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:08.534233 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:08.534238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:08.534242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:08.534727 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:08.535060 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:08.535368 (+ 308us) server_negotiation.cc:299] Negotiation successful
0812 00:58:08.535719 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260812 00:58:08.546732 23266 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:58:08.556247 23266 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:58:08.608716 24049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:08.575562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47088 (local address 127.21.201.66:39205)
0812 00:58:08.596473 (+ 20911us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:08.596482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:08.596514 (+ 32us) server_negotiation.cc:407] Connection header received
0812 00:58:08.596618 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:08.596625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:08.597050 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:08.597303 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:58:08.599205 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.601133 (+ 1928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.604680 (+ 3547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.606493 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.606659 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:08.606998 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:08.608104 (+ 1106us) server_negotiation.cc:299] Negotiation successful
0812 00:58:08.608310 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20645,"thread_start_us":20461,"threads_started":1}
I20260812 00:58:08.608808 24050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:08.574564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:38897 (local address 127.0.0.1:57818)
0812 00:58:08.590129 (+ 15565us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:08.590198 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:08.590455 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:08.598686 (+ 8231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:08.598691 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:08.598720 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:58:08.599258 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:08.599274 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.601993 (+ 2719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.602001 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:08.604476 (+ 2475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:08.604492 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.607245 (+ 2753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.607252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:08.607591 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:08.607762 (+ 171us) client_negotiation.cc:769] Sending connection context
0812 00:58:08.607930 (+ 168us) client_negotiation.cc:240] Negotiation successful
0812 00:58:08.608195 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13092,"mutex_wait_us":100,"thread_start_us":21520,"threads_started":1}
I20260812 00:58:08.610044 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:08.574901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.21.201.65:38897)
0812 00:58:08.598192 (+ 23291us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:08.598199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:08.598221 (+ 22us) server_negotiation.cc:407] Connection header received
0812 00:58:08.598297 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:08.598302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:08.598473 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:08.598692 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:58:08.600125 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.601808 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.604769 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.606486 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.606659 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:08.606998 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:08.609450 (+ 2452us) server_negotiation.cc:299] Negotiation successful
0812 00:58:08.609657 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23098,"thread_start_us":22881,"threads_started":1}
I20260812 00:58:08.610090 24047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:08.575059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:39205 (local address 127.0.0.1:47088)
0812 00:58:08.592367 (+ 17308us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:08.592385 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:08.592486 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:08.597319 (+ 4833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:08.597328 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:08.597395 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:58:08.597943 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:08.597961 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.601329 (+ 3368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.601335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:08.604364 (+ 3029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:08.604399 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.607202 (+ 2803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.607213 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:08.607587 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:08.607762 (+ 175us) client_negotiation.cc:769] Sending connection context
0812 00:58:08.607930 (+ 168us) client_negotiation.cc:240] Negotiation successful
0812 00:58:08.609652 (+ 1722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17180,"mutex_wait_us":1587,"thread_start_us":16997,"threads_started":1}
I20260812 00:58:08.613112 24042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:08.579886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:34297 (local address 127.0.0.1:51034)
0812 00:58:08.596446 (+ 16560us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:08.596497 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:08.596681 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:08.599472 (+ 2791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:08.599477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:08.599507 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:58:08.599907 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:08.599921 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.604102 (+ 4181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.604110 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:08.607044 (+ 2934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:08.607070 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.611973 (+ 4903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.611983 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:08.612248 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:08.612309 (+ 61us) client_negotiation.cc:769] Sending connection context
0812 00:58:08.612422 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 00:58:08.612646 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16376,"thread_start_us":7595,"threads_started":1}
I20260812 00:58:08.614741 24052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:08.580422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51034 (local address 127.21.201.67:34297)
0812 00:58:08.597990 (+ 17568us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:08.597999 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:08.598037 (+ 38us) server_negotiation.cc:407] Connection header received
0812 00:58:08.598147 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:08.598156 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:08.598415 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:08.598663 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:58:08.601581 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.603850 (+ 2269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.607338 (+ 3488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.611262 (+ 3924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.611443 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:08.611785 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:08.613924 (+ 2139us) server_negotiation.cc:299] Negotiation successful
0812 00:58:08.614190 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17350,"thread_start_us":17619,"threads_started":1}
I20260812 00:58:08.614396 23471 tablet_service.cc:1467] Processing CreateTablet for tablet 19f8a5437ff5462289ef3d7ddbdeb61b (DEFAULT_TABLE table=test-table [id=746cb71fed93487ea96f0855f774e27b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:08.616501 23471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f8a5437ff5462289ef3d7ddbdeb61b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:08.617024 23695 tablet_service.cc:1467] Processing CreateTablet for tablet 19f8a5437ff5462289ef3d7ddbdeb61b (DEFAULT_TABLE table=test-table [id=746cb71fed93487ea96f0855f774e27b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:08.618824 23695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f8a5437ff5462289ef3d7ddbdeb61b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:08.619426 23921 tablet_service.cc:1467] Processing CreateTablet for tablet 19f8a5437ff5462289ef3d7ddbdeb61b (DEFAULT_TABLE table=test-table [id=746cb71fed93487ea96f0855f774e27b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:08.623181 23921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f8a5437ff5462289ef3d7ddbdeb61b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:08.641005 24054 tablet_bootstrap.cc:492] T 19f8a5437ff5462289ef3d7ddbdeb61b P 3d256d98a86643909d6d4a1a3ea65ca9: Bootstrap starting.
I20260812 00:58:08.647106 24057 tablet_bootstrap.cc:492] T 19f8a5437ff5462289ef3d7ddbdeb61b P 21abb977fe0747188702c8e263f92661: Bootstrap starting.
I20260812 00:58:08.649189 24054 tablet_bootstrap.cc:654] T 19f8a5437ff5462289ef3d7ddbdeb61b P 3d256d98a86643909d6d4a1a3ea65ca9: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:08.650981 24054 log.cc:824] T 19f8a5437ff5462289ef3d7ddbdeb61b P 3d256d98a86643909d6d4a1a3ea65ca9: Log is configured to *not* fsync() on all Append() calls
I20260812 00:58:08.652436 24058 tablet_bootstrap.cc:492] T 19f8a5437ff5462289ef3d7ddbdeb61b P 8f57d307e1bf478681428ce4a7333aa1: Bootstrap starting.
I20260812 00:58:08.654254 24057 tablet_bootstrap.cc:654] T 19f8a5437ff5462289ef3d7ddbdeb61b P 21abb977fe0747188702c8e263f92661: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:08.656348 24057 log.cc:824] T 19f8a5437ff5462289ef3d7ddbdeb61b P 21abb977fe0747188702c8e263f92661: Log is configured to *not* fsync() on all Append() calls
I20260812 00:58:08.656700 24058 tablet_bootstrap.cc:654] T 19f8a5437ff5462289ef3d7ddbdeb61b P 8f57d307e1bf478681428ce4a7333aa1: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:08.656960 24054 tablet_bootstrap.cc:492] T 19f8a5437ff5462289ef3d7ddbdeb61b P 3d256d98a86643909d6d4a1a3ea65ca9: No bootstrap required, opened a new log
I20260812 00:58:08.657378 24054 ts_tablet_manager.cc:1397] T 19f8a5437ff5462289ef3d7ddbdeb61b P 3d256d98a86643909d6d4a1a3ea65ca9: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s
I20260812 00:58:08.658447 24058 log.cc:824] T 19f8a5437ff5462289ef3d7ddbdeb61b P 8f57d307e1bf478681428ce4a7333aa1: Log is configured to *not* fsync() on all Append() calls
I20260812 00:58:08.667058 24058 tablet_bootstrap.cc:492] T 19f8a5437ff5462289ef3d7ddbdeb61b P 8f57d307e1bf478681428ce4a7333aa1: No bootstrap required, opened a new log
I20260812 00:58:08.667101 24057 tablet_bootstrap.cc:492] T 19f8a5437ff5462289ef3d7ddbdeb61b P 21abb977fe0747188702c8e263f92661: No bootstrap required, opened a new log
I20260812 00:58:08.667531 24058 ts_tablet_manager.cc:1397] T 19f8a5437ff5462289ef3d7ddbdeb61b P 8f57d307e1bf478681428ce4a7333aa1: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.013s
I20260812 00:58:08.667589 24057 ts_tablet_manager.cc:1397] T 19f8a5437ff5462289ef3d7ddbdeb61b P 21abb977fe0747188702c8e263f92661: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.005s
I20260812 00:58:08.672683 24054 raft_consensus.cc:348] T 19f8a5437ff5462289ef3d7ddbdeb61b P 3d256d98a86643909d6d4a1a3ea65ca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d256d98a86643909d6d4a1a3ea65ca9" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 38897 } } peers { permanent_uuid: "21abb977fe0747188702c8e263f92661" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 39205 } } peers { permanent_uuid: "8f57d307e1bf478681428ce4a7333aa1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34297 } }
I20260812 00:58:08.673341 24054 raft_consensus.cc:374] T 19f8a5437ff5462289ef3d7ddbdeb61b P 3d256d98a86643909d6d4a1a3ea65ca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:08.673496 24054 raft_consensus.cc:729] T 19f8a5437ff5462289ef3d7ddbdeb61b P 3d256d98a86643909d6d4a1a3ea65ca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d256d98a86643909d6d4a1a3ea65ca9, State: Initialized, Role: FOLLOWER
I20260812 00:58:08.674345 24054 consensus_queue.cc:260] T 19f8a5437ff5462289ef3d7ddbdeb61b P 3d256d98a86643909d6d4a1a3ea65ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d256d98a86643909d6d4a1a3ea65ca9" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 38897 } } peers { permanent_uuid: "21abb977fe0747188702c8e263f92661" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 39205 } } peers { permanent_uuid: "8f57d307e1bf478681428ce4a7333aa1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34297 } }
I20260812 00:58:08.682233 24058 raft_consensus.cc:348] T 19f8a5437ff5462289ef3d7ddbdeb61b P 8f57d307e1bf478681428ce4a7333aa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d256d98a86643909d6d4a1a3ea65ca9" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 38897 } } peers { permanent_uuid: "21abb977fe0747188702c8e263f92661" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 39205 } } peers { permanent_uuid: "8f57d307e1bf478681428ce4a7333aa1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34297 } }
I20260812 00:58:08.682950 24058 raft_consensus.cc:374] T 19f8a5437ff5462289ef3d7ddbdeb61b P 8f57d307e1bf478681428ce4a7333aa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:08.683133 24058 raft_consensus.cc:729] T 19f8a5437ff5462289ef3d7ddbdeb61b P 8f57d307e1bf478681428ce4a7333aa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f57d307e1bf478681428ce4a7333aa1, State: Initialized, Role: FOLLOWER
I20260812 00:58:08.683359 24057 raft_consensus.cc:348] T 19f8a5437ff5462289ef3d7ddbdeb61b P 21abb977fe0747188702c8e263f92661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d256d98a86643909d6d4a1a3ea65ca9" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 38897 } } peers { permanent_uuid: "21abb977fe0747188702c8e263f92661" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 39205 } } peers { permanent_uuid: "8f57d307e1bf478681428ce4a7333aa1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34297 } }
I20260812 00:58:08.684093 24057 raft_consensus.cc:374] T 19f8a5437ff5462289ef3d7ddbdeb61b P 21abb977fe0747188702c8e263f92661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:08.684284 24057 raft_consensus.cc:729] T 19f8a5437ff5462289ef3d7ddbdeb61b P 21abb977fe0747188702c8e263f92661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21abb977fe0747188702c8e263f92661, State: Initialized, Role: FOLLOWER
I20260812 00:58:08.684229 24058 consensus_queue.cc:260] T 19f8a5437ff5462289ef3d7ddbdeb61b P 8f57d307e1bf478681428ce4a7333aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d256d98a86643909d6d4a1a3ea65ca9" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 38897 } } peers { permanent_uuid: "21abb977fe0747188702c8e263f92661" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 39205 } } peers { permanent_uuid: "8f57d307e1bf478681428ce4a7333aa1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34297 } }
I20260812 00:58:08.685209 24057 consensus_queue.cc:260] T 19f8a5437ff5462289ef3d7ddbdeb61b P 21abb977fe0747188702c8e263f92661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d256d98a86643909d6d4a1a3ea65ca9" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 38897 } } peers { permanent_uuid: "21abb977fe0747188702c8e263f92661" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 39205 } } peers { permanent_uuid: "8f57d307e1bf478681428ce4a7333aa1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34297 } }
I20260812 00:58:08.686460 24054 ts_tablet_manager.cc:1428] T 19f8a5437ff5462289ef3d7ddbdeb61b P 3d256d98a86643909d6d4a1a3ea65ca9: Time spent starting tablet: real 0.029s user 0.015s sys 0.008s
W20260812 00:58:08.692517 23577 tablet.cc:2369] T 19f8a5437ff5462289ef3d7ddbdeb61b P 3d256d98a86643909d6d4a1a3ea65ca9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:58:08.694864 24058 ts_tablet_manager.cc:1428] T 19f8a5437ff5462289ef3d7ddbdeb61b P 8f57d307e1bf478681428ce4a7333aa1: Time spent starting tablet: real 0.027s user 0.019s sys 0.002s
I20260812 00:58:08.695456 24021 heartbeater.cc:503] Master 127.21.201.126:41763 was elected leader, sending a full tablet report...
I20260812 00:58:08.696221 23812 heartbeater.cc:503] Master 127.21.201.126:41763 was elected leader, sending a full tablet report...
I20260812 00:58:08.697489 24057 ts_tablet_manager.cc:1428] T 19f8a5437ff5462289ef3d7ddbdeb61b P 21abb977fe0747188702c8e263f92661: Time spent starting tablet: real 0.030s user 0.023s sys 0.005s
W20260812 00:58:08.716503 24022 tablet.cc:2369] T 19f8a5437ff5462289ef3d7ddbdeb61b P 8f57d307e1bf478681428ce4a7333aa1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:58:08.765445 24067 raft_consensus.cc:482] T 19f8a5437ff5462289ef3d7ddbdeb61b P 21abb977fe0747188702c8e263f92661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:58:08.765832 24067 raft_consensus.cc:504] T 19f8a5437ff5462289ef3d7ddbdeb61b P 21abb977fe0747188702c8e263f92661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d256d98a86643909d6d4a1a3ea65ca9" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 38897 } } peers { permanent_uuid: "21abb977fe0747188702c8e263f92661" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 39205 } } peers { permanent_uuid: "8f57d307e1bf478681428ce4a7333aa1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34297 } }
I20260812 00:58:08.768110 24067 leader_election.cc:288] T 19f8a5437ff5462289ef3d7ddbdeb61b P 21abb977fe0747188702c8e263f92661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d256d98a86643909d6d4a1a3ea65ca9 (127.21.201.65:38897), 8f57d307e1bf478681428ce4a7333aa1 (127.21.201.67:34297)
I20260812 00:58:08.777719 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:08.769277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:51753 (local address 127.21.201.65:38897)
0812 00:58:08.769525 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:08.769534 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:08.770662 (+ 1128us) server_negotiation.cc:407] Connection header received
0812 00:58:08.770931 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:08.770939 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:08.771049 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:08.771151 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:58:08.772085 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.773076 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.775387 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.776623 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.776893 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:08.776993 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:08.777295 (+ 302us) server_negotiation.cc:299] Negotiation successful
0812 00:58:08.777433 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 00:58:08.778028 24069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:08.768598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:38897 (local address 127.21.201.66:51753)
0812 00:58:08.770502 (+ 1904us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:08.770526 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:08.770734 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:08.771211 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:08.771219 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:08.771255 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:58:08.771899 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:08.771917 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.773228 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.773234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:08.775193 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:08.775209 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.776920 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.776928 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:08.777121 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:08.777146 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 00:58:08.777267 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 00:58:08.777422 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1727,"mutex_wait_us":4,"thread_start_us":925,"threads_started":1}
I20260812 00:58:08.779167 24052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:08.768652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:33949 (local address 127.21.201.67:34297)
0812 00:58:08.768921 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:08.768930 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:08.769776 (+ 846us) server_negotiation.cc:407] Connection header received
0812 00:58:08.770157 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:08.770163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:08.770248 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:08.770316 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:58:08.771300 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.772265 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.775890 (+ 3625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.777593 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.777787 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:08.777894 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:08.778728 (+ 834us) server_negotiation.cc:299] Negotiation successful
0812 00:58:08.778891 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 00:58:08.779518 24071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:08.768425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:34297 (local address 127.21.201.66:33949)
0812 00:58:08.769612 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:08.769644 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:08.769942 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:08.770500 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:08.770504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:08.770526 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:58:08.771115 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:08.771132 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.772478 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.772486 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:08.775630 (+ 3144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:08.775651 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.778118 (+ 2467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.778128 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:08.778519 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:08.778566 (+ 47us) client_negotiation.cc:769] Sending connection context
0812 00:58:08.778678 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 00:58:08.778857 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1005,"mutex_wait_us":62,"spinlock_wait_cycles":22272,"thread_start_us":1616,"threads_started":1}
I20260812 00:58:08.779359 23504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f8a5437ff5462289ef3d7ddbdeb61b" candidate_uuid: "21abb977fe0747188702c8e263f92661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d256d98a86643909d6d4a1a3ea65ca9" is_pre_election: true
I20260812 00:58:08.780087 23504 raft_consensus.cc:2454] T 19f8a5437ff5462289ef3d7ddbdeb61b P 3d256d98a86643909d6d4a1a3ea65ca9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21abb977fe0747188702c8e263f92661 in term 0.
I20260812 00:58:08.780794 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f8a5437ff5462289ef3d7ddbdeb61b" candidate_uuid: "21abb977fe0747188702c8e263f92661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f57d307e1bf478681428ce4a7333aa1" is_pre_election: true
I20260812 00:58:08.780983 23606 leader_election.cc:302] T 19f8a5437ff5462289ef3d7ddbdeb61b P 21abb977fe0747188702c8e263f92661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21abb977fe0747188702c8e263f92661, 3d256d98a86643909d6d4a1a3ea65ca9; no voters:
I20260812 00:58:08.781409 23949 raft_consensus.cc:2454] T 19f8a5437ff5462289ef3d7ddbdeb61b P 8f57d307e1bf478681428ce4a7333aa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21abb977fe0747188702c8e263f92661 in term 0.
I20260812 00:58:08.781919 24067 raft_consensus.cc:2781] T 19f8a5437ff5462289ef3d7ddbdeb61b P 21abb977fe0747188702c8e263f92661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:58:08.782094 24067 raft_consensus.cc:482] T 19f8a5437ff5462289ef3d7ddbdeb61b P 21abb977fe0747188702c8e263f92661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:58:08.782208 24067 raft_consensus.cc:3037] T 19f8a5437ff5462289ef3d7ddbdeb61b P 21abb977fe0747188702c8e263f92661 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:08.784984 24067 raft_consensus.cc:504] T 19f8a5437ff5462289ef3d7ddbdeb61b P 21abb977fe0747188702c8e263f92661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d256d98a86643909d6d4a1a3ea65ca9" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 38897 } } peers { permanent_uuid: "21abb977fe0747188702c8e263f92661" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 39205 } } peers { permanent_uuid: "8f57d307e1bf478681428ce4a7333aa1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34297 } }
I20260812 00:58:08.785903 24067 leader_election.cc:288] T 19f8a5437ff5462289ef3d7ddbdeb61b P 21abb977fe0747188702c8e263f92661 [CANDIDATE]: Term 1 election: Requested vote from peers 3d256d98a86643909d6d4a1a3ea65ca9 (127.21.201.65:38897), 8f57d307e1bf478681428ce4a7333aa1 (127.21.201.67:34297)
I20260812 00:58:08.786463 23504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f8a5437ff5462289ef3d7ddbdeb61b" candidate_uuid: "21abb977fe0747188702c8e263f92661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d256d98a86643909d6d4a1a3ea65ca9"
I20260812 00:58:08.786607 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f8a5437ff5462289ef3d7ddbdeb61b" candidate_uuid: "21abb977fe0747188702c8e263f92661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f57d307e1bf478681428ce4a7333aa1"
I20260812 00:58:08.786734 23504 raft_consensus.cc:3037] T 19f8a5437ff5462289ef3d7ddbdeb61b P 3d256d98a86643909d6d4a1a3ea65ca9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:08.786921 23949 raft_consensus.cc:3037] T 19f8a5437ff5462289ef3d7ddbdeb61b P 8f57d307e1bf478681428ce4a7333aa1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:08.790408 23504 raft_consensus.cc:2454] T 19f8a5437ff5462289ef3d7ddbdeb61b P 3d256d98a86643909d6d4a1a3ea65ca9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21abb977fe0747188702c8e263f92661 in term 1.
I20260812 00:58:08.791045 23606 leader_election.cc:302] T 19f8a5437ff5462289ef3d7ddbdeb61b P 21abb977fe0747188702c8e263f92661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21abb977fe0747188702c8e263f92661, 3d256d98a86643909d6d4a1a3ea65ca9; no voters:
I20260812 00:58:08.791419 24067 raft_consensus.cc:2781] T 19f8a5437ff5462289ef3d7ddbdeb61b P 21abb977fe0747188702c8e263f92661 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:58:08.792243 24067 raft_consensus.cc:686] T 19f8a5437ff5462289ef3d7ddbdeb61b P 21abb977fe0747188702c8e263f92661 [term 1 LEADER]: Becoming Leader. State: Replica: 21abb977fe0747188702c8e263f92661, State: Running, Role: LEADER
I20260812 00:58:08.792999 23949 raft_consensus.cc:2454] T 19f8a5437ff5462289ef3d7ddbdeb61b P 8f57d307e1bf478681428ce4a7333aa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21abb977fe0747188702c8e263f92661 in term 1.
I20260812 00:58:08.792893 24067 consensus_queue.cc:237] T 19f8a5437ff5462289ef3d7ddbdeb61b P 21abb977fe0747188702c8e263f92661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d256d98a86643909d6d4a1a3ea65ca9" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 38897 } } peers { permanent_uuid: "21abb977fe0747188702c8e263f92661" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 39205 } } peers { permanent_uuid: "8f57d307e1bf478681428ce4a7333aa1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34297 } }
I20260812 00:58:08.798978 23266 catalog_manager.cc:5495] T 19f8a5437ff5462289ef3d7ddbdeb61b P 21abb977fe0747188702c8e263f92661 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21abb977fe0747188702c8e263f92661 (127.21.201.66). New cstate: current_term: 1 leader_uuid: "21abb977fe0747188702c8e263f92661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d256d98a86643909d6d4a1a3ea65ca9" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 38897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21abb977fe0747188702c8e263f92661" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 39205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f57d307e1bf478681428ce4a7333aa1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34297 } health_report { overall_health: UNKNOWN } } }
W20260812 00:58:08.816926 23813 tablet.cc:2369] T 19f8a5437ff5462289ef3d7ddbdeb61b P 21abb977fe0747188702c8e263f92661: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 00:58:08 dist-test-slave-7f01 krb5kdc[23202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.65@KRBTEST.COM
I20260812 00:58:08.851508 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:08.826061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57826 (local address 127.21.201.65:38897)
0812 00:58:08.826323 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:08.826332 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:08.826973 (+ 641us) server_negotiation.cc:407] Connection header received
0812 00:58:08.827151 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:08.827158 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:08.827299 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:08.827468 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:08.829967 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.831514 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.833483 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.834462 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.834686 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:08.840956 (+ 6270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:08.840988 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:08.841015 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:08.841085 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:08.846644 (+ 5559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:08.847804 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:08.847818 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:08.847824 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:08.848080 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:08.848921 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:08.848929 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:08.848934 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:08.849632 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:08.850049 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:08.850588 (+ 539us) server_negotiation.cc:299] Negotiation successful
0812 00:58:08.850936 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Aug 12 00:58:08 dist-test-slave-7f01 krb5kdc[23202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496288, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 00:58:08 dist-test-slave-7f01 krb5kdc[23202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496288, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.201.65@KRBTEST.COM
I20260812 00:58:08.923995 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:08.903664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57832 (local address 127.21.201.65:38897)
0812 00:58:08.903984 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:08.903994 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:08.904561 (+ 567us) server_negotiation.cc:407] Connection header received
0812 00:58:08.904852 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:08.904863 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:08.904948 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:08.905076 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:08.906809 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.907664 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.909652 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.910180 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.910373 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:08.916338 (+ 5965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:08.916350 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:08.916357 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:08.916434 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:08.919948 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:08.921220 (+ 1272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:08.921227 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:08.921229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:08.921443 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:08.921964 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:08.921969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:08.921972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:08.922359 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:08.922763 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:08.923220 (+ 457us) server_negotiation.cc:299] Negotiation successful
0812 00:58:08.923532 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
Aug 12 00:58:08 dist-test-slave-7f01 krb5kdc[23202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 00:58:08 dist-test-slave-7f01 krb5kdc[23202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.65@KRBTEST.COM
I20260812 00:58:08.991889 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:08.971043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.21.201.65:38897)
0812 00:58:08.971335 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:08.971345 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:08.972036 (+ 691us) server_negotiation.cc:407] Connection header received
0812 00:58:08.972285 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:08.972290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:08.972366 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:08.972490 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:08.974297 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.975039 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.976612 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:08.977107 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:08.977332 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:08.982848 (+ 5516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:08.982857 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:08.982862 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:08.982945 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:08.987504 (+ 4559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:08.988861 (+ 1357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:08.988868 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:08.988871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:08.989118 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:08.989853 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:08.989857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:08.989859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:08.990251 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:08.990490 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:08.991065 (+ 575us) server_negotiation.cc:299] Negotiation successful
0812 00:58:08.991338 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
W20260812 00:58:08.994422 23441 tablet_service.cc:2908] Rejecting scan request for tablet 19f8a5437ff5462289ef3d7ddbdeb61b: Uninitialized: clean time has not yet been initialized
I20260812 00:58:08.999270 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 23371
I20260812 00:58:09.019600 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:09.001311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57848 (local address 127.21.201.65:38897)
0812 00:58:09.001525 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:09.001530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:09.001995 (+ 465us) server_negotiation.cc:407] Connection header received
0812 00:58:09.002318 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:09.002323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:09.002442 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:09.002553 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:09.004505 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:09.005316 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:09.007047 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:09.007602 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:09.007966 (+ 364us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:09.011147 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:09.011155 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:09.011159 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:09.011214 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:09.015288 (+ 4074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:09.016482 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:09.016489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:09.016492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:09.016781 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:09.017622 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:09.017627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:09.017629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:09.018051 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:09.018378 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:09.019061 (+ 683us) server_negotiation.cc:299] Negotiation successful
0812 00:58:09.019317 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 00:58:09.020503 23569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:09.122563 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 23371
I20260812 00:58:09.152503 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 23583
Aug 12 00:58:09 dist-test-slave-7f01 krb5kdc[23202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.66@KRBTEST.COM
I20260812 00:58:09.173395 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:09.153634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47100 (local address 127.21.201.66:39205)
0812 00:58:09.154206 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:09.154210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:09.154229 (+ 19us) server_negotiation.cc:407] Connection header received
0812 00:58:09.154290 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:09.154294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:09.154358 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:09.154476 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:09.156075 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:09.157015 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:09.158576 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:09.159326 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:09.159457 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:09.165157 (+ 5700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:09.165185 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:09.165207 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:09.165266 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:09.169400 (+ 4134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:09.170793 (+ 1393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:09.170804 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:09.170806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:09.171073 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:09.171492 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:09.171496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:09.171498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:09.172207 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:09.172491 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:09.172866 (+ 375us) server_negotiation.cc:299] Negotiation successful
0812 00:58:09.173122 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":302,"threads_started":1}
I20260812 00:58:09.174549 23801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:09.274636 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 23583
I20260812 00:58:09.311558 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 23818
Aug 12 00:58:09 dist-test-slave-7f01 krb5kdc[23202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.67@KRBTEST.COM
I20260812 00:58:09.332676 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:09.312674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51044 (local address 127.21.201.67:34297)
0812 00:58:09.313588 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:09.313594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:09.313614 (+ 20us) server_negotiation.cc:407] Connection header received
0812 00:58:09.313677 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:09.313681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:09.313759 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:09.313871 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:09.315642 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:09.316536 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:09.318134 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:09.318863 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:09.319044 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:09.324626 (+ 5582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:09.324650 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:09.324674 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:09.324736 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:09.328372 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:09.329656 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:09.329669 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:09.329672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:09.330043 (+ 371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:09.330724 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:09.330728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:09.330731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:09.331221 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:09.331536 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:09.331961 (+ 425us) server_negotiation.cc:299] Negotiation successful
0812 00:58:09.332201 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":520,"threads_started":1}
I20260812 00:58:09.333555 24013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:09.430894 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 23818
I20260812 00:58:09.456893 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 23221
I20260812 00:58:09.458786 23316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:09.577495 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 23221
2026-08-12T00:58:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (5509 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:58:09 dist-test-slave-7f01 krb5kdc[24098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:58:09 dist-test-slave-7f01 krb5kdc[24098](info): set up 2 sockets
Aug 12 00:58:09 dist-test-slave-7f01 krb5kdc[24098](info): commencing operation
krb5kdc: starting...
W20260812 00:58:11.056527 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.416s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 00:58:11 dist-test-slave-7f01 krb5kdc[24098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:58:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:58:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 00:58:11.313531 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:34651
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:34651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:11.705240 24118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:11.705705 24118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:11.705818 24118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:11.715329 24118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:58:11.715562 24118 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 00:58:11.715612 24118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:11.715679 24118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:11.716043 24118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:58:11.716145 24118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:58:11.728754 24118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:34651
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:34651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:11.730310 24118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:11.731902 24118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:11.748412 24125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:11.748561 24126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:11.748469 24128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:11.750571 24118 server_base.cc:1034] running on GCE node
I20260812 00:58:11.751725 24118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:11.753636 24118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:11.754940 24118 hybrid_clock.cc:648] HybridClock initialized: now 1786496291754859 us; error 37 us; skew 500 ppm
I20260812 00:58:11.755435 24118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:11 dist-test-slave-7f01 krb5kdc[24098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496291, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:11.764885 24118 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 00:58:11.767447 24118 webserver.cc:460] Webserver started at http://127.21.201.126:35571/ using document root <none> and password file <none>
I20260812 00:58:11.768630 24118 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:11.768810 24118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:11.769256 24118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:11.774544 24118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "caf58d56f64846f6a93c0e452d73e974"
format_stamp: "Formatted at 2026-08-12 00:58:11 on dist-test-slave-7f01"
server_key: "3d50836cb3bf44e0a5865667c63f4bc7"
server_key_iv: "78a130f59d8b2f9a3edffbcebcfcd892"
server_key_version: "encryptionkey@0"
I20260812 00:58:11.776348 24118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "caf58d56f64846f6a93c0e452d73e974"
format_stamp: "Formatted at 2026-08-12 00:58:11 on dist-test-slave-7f01"
server_key: "3d50836cb3bf44e0a5865667c63f4bc7"
server_key_iv: "78a130f59d8b2f9a3edffbcebcfcd892"
server_key_version: "encryptionkey@0"
I20260812 00:58:11.785323 24118 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20260812 00:58:11.790290 24148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:11.792196 24118 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260812 00:58:11.792454 24118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "caf58d56f64846f6a93c0e452d73e974"
format_stamp: "Formatted at 2026-08-12 00:58:11 on dist-test-slave-7f01"
server_key: "3d50836cb3bf44e0a5865667c63f4bc7"
server_key_iv: "78a130f59d8b2f9a3edffbcebcfcd892"
server_key_version: "encryptionkey@0"
I20260812 00:58:11.792742 24118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:11.820317 24118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:11.827565 24118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:11.828078 24118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:11.858018 24118 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:34651
I20260812 00:58:11.858121 24246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:34651 every 8 connection(s)
I20260812 00:58:11.860184 24118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 00:58:11.864378 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 24118
I20260812 00:58:11.864712 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 00:58:11.865520 22309 external_mini_cluster.cc:1428] Setting key 177aa94699956eca8fac7c4dec1561ed
I20260812 00:58:11.866137 24247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:11.890133 24247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caf58d56f64846f6a93c0e452d73e974: Bootstrap starting.
Aug 12 00:58:11 dist-test-slave-7f01 krb5kdc[24098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:11.894572 24247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P caf58d56f64846f6a93c0e452d73e974: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:11.896337 24247 log.cc:824] T 00000000000000000000000000000000 P caf58d56f64846f6a93c0e452d73e974: Log is configured to *not* fsync() on all Append() calls
I20260812 00:58:11.901124 24247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caf58d56f64846f6a93c0e452d73e974: No bootstrap required, opened a new log
I20260812 00:58:11.903059 24253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:11.868522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58432 (local address 127.21.201.126:34651)
0812 00:58:11.869731 (+ 1209us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:11.869747 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:11.869807 (+ 60us) server_negotiation.cc:407] Connection header received
0812 00:58:11.873691 (+ 3884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:11.873734 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:11.874580 (+ 846us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:11.875256 (+ 676us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:11.877938 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:11.880796 (+ 2858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:11.883215 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:11.884634 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:11.884901 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:11.892679 (+ 7778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:11.892693 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:11.892716 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:11.892786 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:11.898276 (+ 5490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:11.899253 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:11.899261 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:11.899268 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:11.899463 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:11.899965 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:11.899968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:11.899970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:11.900547 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:11.900877 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:11.901360 (+ 483us) server_negotiation.cc:299] Negotiation successful
0812 00:58:11.901894 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":911,"thread_start_us":594,"threads_started":1}
I20260812 00:58:11.915055 24247 raft_consensus.cc:348] T 00000000000000000000000000000000 P caf58d56f64846f6a93c0e452d73e974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf58d56f64846f6a93c0e452d73e974" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 34651 } }
I20260812 00:58:11.915854 24247 raft_consensus.cc:374] T 00000000000000000000000000000000 P caf58d56f64846f6a93c0e452d73e974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:11.915995 24247 raft_consensus.cc:729] T 00000000000000000000000000000000 P caf58d56f64846f6a93c0e452d73e974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caf58d56f64846f6a93c0e452d73e974, State: Initialized, Role: FOLLOWER
I20260812 00:58:11.916723 24247 consensus_queue.cc:260] T 00000000000000000000000000000000 P caf58d56f64846f6a93c0e452d73e974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf58d56f64846f6a93c0e452d73e974" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 34651 } }
I20260812 00:58:11.917011 24247 raft_consensus.cc:388] T 00000000000000000000000000000000 P caf58d56f64846f6a93c0e452d73e974 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:58:11.917167 24247 raft_consensus.cc:482] T 00000000000000000000000000000000 P caf58d56f64846f6a93c0e452d73e974 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:58:11.917387 24247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P caf58d56f64846f6a93c0e452d73e974 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:11.920404 24247 raft_consensus.cc:504] T 00000000000000000000000000000000 P caf58d56f64846f6a93c0e452d73e974 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf58d56f64846f6a93c0e452d73e974" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 34651 } }
I20260812 00:58:11.921084 24247 leader_election.cc:302] T 00000000000000000000000000000000 P caf58d56f64846f6a93c0e452d73e974 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: caf58d56f64846f6a93c0e452d73e974; no voters:
I20260812 00:58:11.921940 24247 leader_election.cc:288] T 00000000000000000000000000000000 P caf58d56f64846f6a93c0e452d73e974 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:58:11.922271 24258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P caf58d56f64846f6a93c0e452d73e974 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:58:11.923621 24258 raft_consensus.cc:686] T 00000000000000000000000000000000 P caf58d56f64846f6a93c0e452d73e974 [term 1 LEADER]: Becoming Leader. State: Replica: caf58d56f64846f6a93c0e452d73e974, State: Running, Role: LEADER
I20260812 00:58:11.924583 24258 consensus_queue.cc:237] T 00000000000000000000000000000000 P caf58d56f64846f6a93c0e452d73e974 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf58d56f64846f6a93c0e452d73e974" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 34651 } }
I20260812 00:58:11.924763 24247 sys_catalog.cc:564] T 00000000000000000000000000000000 P caf58d56f64846f6a93c0e452d73e974 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:58:11.930112 24259 sys_catalog.cc:455] T 00000000000000000000000000000000 P caf58d56f64846f6a93c0e452d73e974 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "caf58d56f64846f6a93c0e452d73e974" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf58d56f64846f6a93c0e452d73e974" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 34651 } } }
I20260812 00:58:11.930651 24259 sys_catalog.cc:458] T 00000000000000000000000000000000 P caf58d56f64846f6a93c0e452d73e974 [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:11.930056 24261 sys_catalog.cc:455] T 00000000000000000000000000000000 P caf58d56f64846f6a93c0e452d73e974 [sys.catalog]: SysCatalogTable state changed. Reason: New leader caf58d56f64846f6a93c0e452d73e974. Latest consensus state: current_term: 1 leader_uuid: "caf58d56f64846f6a93c0e452d73e974" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf58d56f64846f6a93c0e452d73e974" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 34651 } } }
I20260812 00:58:11.930855 24261 sys_catalog.cc:458] T 00000000000000000000000000000000 P caf58d56f64846f6a93c0e452d73e974 [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:11.933648 24271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:58:11.940490 24271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:58:11.953912 24271 catalog_manager.cc:1347] Generated new cluster ID: d2683e9ff1b44da193f67da44293dfb8
I20260812 00:58:11.954114 24271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:58:11.974028 24271 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:58:11.975556 24271 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:58:11.994488 24271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P caf58d56f64846f6a93c0e452d73e974: Generated new TSK 0
I20260812 00:58:11.995518 24271 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.201.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
I20260812 00:58:12.065497 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:0
--local_ip_for_outbound_sockets=127.21.201.65
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:34651
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:12.449138 24296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:12.449703 24296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:12.449904 24296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:12.459245 24296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:12.459506 24296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:12.459669 24296 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:58:12.459851 24296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 00:58:12.471248 24296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.201.126:34651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:12.472891 24296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:12.474445 24296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:12.492571 24306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:12.492476 24307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:12.492594 24309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:12.493976 24296 server_base.cc:1034] running on GCE node
I20260812 00:58:12.495208 24296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:12.497006 24296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:12.498375 24296 hybrid_clock.cc:648] HybridClock initialized: now 1786496292498259 us; error 77 us; skew 500 ppm
I20260812 00:58:12.498859 24296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:12 dist-test-slave-7f01 krb5kdc[24098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496292, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:12.503844 24296 init.cc:377] Logged in from keytab as kudu/127.21.201.65@KRBTEST.COM (short username kudu)
I20260812 00:58:12.506420 24296 webserver.cc:460] Webserver started at http://127.21.201.65:33469/ using document root <none> and password file <none>
I20260812 00:58:12.507532 24296 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:12.507704 24296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:12.508216 24296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:12.512452 24296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-0/data/instance:
uuid: "35285f43f5584e589f0f1f36b5e04aee"
format_stamp: "Formatted at 2026-08-12 00:58:12 on dist-test-slave-7f01"
server_key: "228f21f6d5bfb1ec583f3c11a8fa0560"
server_key_iv: "65542147ac01d85145a0df1dc54aafe9"
server_key_version: "encryptionkey@0"
I20260812 00:58:12.513587 24296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance:
uuid: "35285f43f5584e589f0f1f36b5e04aee"
format_stamp: "Formatted at 2026-08-12 00:58:12 on dist-test-slave-7f01"
server_key: "228f21f6d5bfb1ec583f3c11a8fa0560"
server_key_iv: "65542147ac01d85145a0df1dc54aafe9"
server_key_version: "encryptionkey@0"
I20260812 00:58:12.521538 24296 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 00:58:12.527614 24324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:12.529807 24296 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260812 00:58:12.530113 24296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "35285f43f5584e589f0f1f36b5e04aee"
format_stamp: "Formatted at 2026-08-12 00:58:12 on dist-test-slave-7f01"
server_key: "228f21f6d5bfb1ec583f3c11a8fa0560"
server_key_iv: "65542147ac01d85145a0df1dc54aafe9"
server_key_version: "encryptionkey@0"
I20260812 00:58:12.530445 24296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:12.579634 24296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:12.586131 24296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:12.586516 24296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:12.590400 24296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:12.590565 24296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:12.590759 24296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:12.590853 24296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 00:58:12 dist-test-slave-7f01 krb5kdc[24098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496292, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:12.626966 24339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:12.591547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:43105 (local address 127.21.201.126:34651)
0812 00:58:12.592397 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:12.592405 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:12.594482 (+ 2077us) server_negotiation.cc:407] Connection header received
0812 00:58:12.598820 (+ 4338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:12.598826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:12.598936 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:12.599134 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:12.603159 (+ 4025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:12.604486 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:12.607783 (+ 3297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:12.608698 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:12.608923 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:12.615021 (+ 6098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:12.615033 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:12.615038 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:12.615126 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:12.619955 (+ 4829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:12.621838 (+ 1883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:12.621847 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:12.621854 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:12.622125 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:12.623162 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:12.623171 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:12.623178 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:12.623665 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:12.623894 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:12.625821 (+ 1927us) server_negotiation.cc:299] Negotiation successful
0812 00:58:12.626286 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":513,"threads_started":1}
I20260812 00:58:12.628501 24340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:12.592796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:34651 (local address 127.21.201.65:43105)
0812 00:58:12.594144 (+ 1348us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:12.594248 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:12.598199 (+ 3951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:12.599625 (+ 1426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:12.599644 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:12.601055 (+ 1411us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:12.602678 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:12.602731 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:12.604757 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:12.604766 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:12.607381 (+ 2615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:12.607403 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:12.609164 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:12.609179 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:12.609597 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:12.610868 (+ 1271us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:12.610902 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:12.614622 (+ 3720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:12.620307 (+ 5685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:12.620330 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:12.620354 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:12.621541 (+ 1187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:12.622352 (+ 811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:12.622363 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:12.622371 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:12.622947 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:12.624004 (+ 1057us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:12.624021 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:12.624878 (+ 857us) client_negotiation.cc:769] Sending connection context
0812 00:58:12.625636 (+ 758us) client_negotiation.cc:240] Negotiation successful
0812 00:58:12.626562 (+ 926us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"thread_start_us":515,"threads_started":1}
I20260812 00:58:12.653481 24296 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:36177
I20260812 00:58:12.653649 24523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:36177 every 8 connection(s)
I20260812 00:58:12.655315 24296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 00:58:12.657687 24339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:12.638795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:60993 (local address 127.21.201.126:34651)
0812 00:58:12.639071 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:12.639079 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:12.639813 (+ 734us) server_negotiation.cc:407] Connection header received
0812 00:58:12.640125 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:12.640133 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:12.640243 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:12.640398 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:12.641526 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:12.643102 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:12.645142 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:12.645673 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:12.645829 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:12.648604 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:12.648614 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:12.648621 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:12.648685 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:12.653161 (+ 4476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:12.654321 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:12.654328 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:12.654331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:12.654524 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:12.655000 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:12.655005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:12.655008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:12.655467 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:12.655678 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:12.656759 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0812 00:58:12.657258 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260812 00:58:12.658262 24485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:12.638801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:34651 (local address 127.21.201.65:60993)
0812 00:58:12.639640 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:12.639677 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:12.639946 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:12.640437 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:12.640443 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:12.641004 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:12.641332 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:12.641352 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:12.643336 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:12.643342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:12.644853 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:12.644871 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:12.645924 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:12.645932 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:12.646113 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:12.646983 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:12.647016 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:12.648110 (+ 1094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:12.653506 (+ 5396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:12.653515 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:12.653519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:12.654140 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:12.654673 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:12.654677 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:12.654680 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:12.654854 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:12.655666 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:12.655670 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:12.655992 (+ 322us) client_negotiation.cc:769] Sending connection context
0812 00:58:12.656478 (+ 486us) client_negotiation.cc:240] Negotiation successful
0812 00:58:12.657543 (+ 1065us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":458,"threads_started":1}
I20260812 00:58:12.659003 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 24296
I20260812 00:58:12.659276 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance
I20260812 00:58:12.660063 22309 external_mini_cluster.cc:1428] Setting key 08a50bdcff959bc67215163b82d02f4a
I20260812 00:58:12.660742 24524 heartbeater.cc:347] Connected to a master server at 127.21.201.126:34651
I20260812 00:58:12.661095 24524 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:12.662501 24524 heartbeater.cc:511] Master 127.21.201.126:34651 requested a full tablet report, sending...
I20260812 00:58:12.664995 24176 ts_manager.cc:194] Registered new tserver with Master: 35285f43f5584e589f0f1f36b5e04aee (127.21.201.65:36177)
I20260812 00:58:12.666594 24176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.65@KRBTEST.COM'} at 127.21.201.65:43105
WARNING: no policy specified for kudu/127.21.201.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
I20260812 00:58:12.717571 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:0
--local_ip_for_outbound_sockets=127.21.201.66
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:34651
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:13.106386 24531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:13.106760 24531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:13.106971 24531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:13.115015 24531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:13.115175 24531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:13.115329 24531 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:58:13.115492 24531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 00:58:13.126433 24531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.201.126:34651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:13.128059 24531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:13.129613 24531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:13.145391 24540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:13.145390 24542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:13.145524 24545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:13.146368 24531 server_base.cc:1034] running on GCE node
I20260812 00:58:13.147186 24531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:13.148432 24531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:13.149761 24531 hybrid_clock.cc:648] HybridClock initialized: now 1786496293149683 us; error 59 us; skew 500 ppm
I20260812 00:58:13.150180 24531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:13 dist-test-slave-7f01 krb5kdc[24098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496293, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:13.154212 24531 init.cc:377] Logged in from keytab as kudu/127.21.201.66@KRBTEST.COM (short username kudu)
I20260812 00:58:13.156323 24531 webserver.cc:460] Webserver started at http://127.21.201.66:41739/ using document root <none> and password file <none>
I20260812 00:58:13.157187 24531 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:13.157346 24531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:13.157734 24531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:13.161509 24531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-1/data/instance:
uuid: "a6d1a209f49341ee827da80c645e69db"
format_stamp: "Formatted at 2026-08-12 00:58:13 on dist-test-slave-7f01"
server_key: "c86c796d96699d32708d3fdd1c4ff5a0"
server_key_iv: "2f7f6fdeefcca062e25fc3578eeb0bf6"
server_key_version: "encryptionkey@0"
I20260812 00:58:13.162595 24531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance:
uuid: "a6d1a209f49341ee827da80c645e69db"
format_stamp: "Formatted at 2026-08-12 00:58:13 on dist-test-slave-7f01"
server_key: "c86c796d96699d32708d3fdd1c4ff5a0"
server_key_iv: "2f7f6fdeefcca062e25fc3578eeb0bf6"
server_key_version: "encryptionkey@0"
I20260812 00:58:13.169914 24531 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260812 00:58:13.174467 24555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:13.176468 24531 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 00:58:13.176709 24531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "a6d1a209f49341ee827da80c645e69db"
format_stamp: "Formatted at 2026-08-12 00:58:13 on dist-test-slave-7f01"
server_key: "c86c796d96699d32708d3fdd1c4ff5a0"
server_key_iv: "2f7f6fdeefcca062e25fc3578eeb0bf6"
server_key_version: "encryptionkey@0"
I20260812 00:58:13.177003 24531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:13.205945 24531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:13.212165 24531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:13.212532 24531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:13.216032 24531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:13.216174 24531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:13.216348 24531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:13.216439 24531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 00:58:13 dist-test-slave-7f01 krb5kdc[24098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496293, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:13.244588 24566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:13.216732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:40005 (local address 127.21.201.126:34651)
0812 00:58:13.217488 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:13.217495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:13.218924 (+ 1429us) server_negotiation.cc:407] Connection header received
0812 00:58:13.221847 (+ 2923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:13.221857 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:13.221994 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:13.222148 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:13.224932 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:13.226137 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:13.228150 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:13.228886 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:13.229038 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:13.235509 (+ 6471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:13.235519 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:13.235525 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:13.235580 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:13.240786 (+ 5206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:13.241785 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:13.241791 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:13.241795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:13.241958 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:13.242472 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:13.242477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:13.242481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:13.242793 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:13.242979 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:13.243850 (+ 871us) server_negotiation.cc:299] Negotiation successful
0812 00:58:13.244139 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":487,"threads_started":1}
I20260812 00:58:13.247018 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:13.217601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:34651 (local address 127.21.201.66:40005)
0812 00:58:13.218688 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:13.218743 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:13.221292 (+ 2549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:13.222526 (+ 1234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:13.222545 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:13.223502 (+ 957us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:13.224586 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:13.224610 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:13.226329 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:13.226334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:13.227910 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:13.227929 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:13.229209 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:13.229221 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:13.229445 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:13.230340 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:13.230392 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:13.235057 (+ 4665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:13.241005 (+ 5948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:13.241018 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:13.241042 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:13.241626 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:13.242092 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:13.242097 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:13.242101 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:13.242339 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:13.242949 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:13.242959 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:13.243389 (+ 430us) client_negotiation.cc:769] Sending connection context
0812 00:58:13.243693 (+ 304us) client_negotiation.cc:240] Negotiation successful
0812 00:58:13.244461 (+ 768us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"thread_start_us":480,"threads_started":1}
I20260812 00:58:13.275391 24566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:13.256804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:46041 (local address 127.21.201.126:34651)
0812 00:58:13.257111 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:13.257121 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:13.257682 (+ 561us) server_negotiation.cc:407] Connection header received
0812 00:58:13.257849 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:13.257865 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:13.257989 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:13.258160 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:13.259342 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:13.260747 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:13.263287 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:13.263976 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:13.264149 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:13.268328 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:13.268335 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:13.268339 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:13.268385 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:13.271807 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:13.273134 (+ 1327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:13.273143 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:13.273147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:13.273304 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:13.273709 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:13.273714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:13.273719 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:13.274106 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:13.274285 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:13.274718 (+ 433us) server_negotiation.cc:299] Negotiation successful
0812 00:58:13.274996 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260812 00:58:13.275509 24674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:13.256798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:34651 (local address 127.21.201.66:46041)
0812 00:58:13.257539 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:13.257562 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:13.257721 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:13.258221 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:13.258225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:13.258802 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:13.259148 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:13.259159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:13.261014 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:13.261025 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:13.262863 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:13.262896 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:13.264507 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:13.264520 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:13.264842 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:13.266333 (+ 1491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:13.266359 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:13.267961 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:13.272041 (+ 4080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:13.272048 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:13.272053 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:13.272973 (+ 920us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:13.273439 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:13.273448 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:13.273451 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:13.273592 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:13.274248 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:13.274254 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:13.274458 (+ 204us) client_negotiation.cc:769] Sending connection context
0812 00:58:13.274617 (+ 159us) client_negotiation.cc:240] Negotiation successful
0812 00:58:13.275003 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":397,"threads_started":1}
I20260812 00:58:13.289117 24531 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:40765
I20260812 00:58:13.289170 24757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:40765 every 8 connection(s)
I20260812 00:58:13.290618 24531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 00:58:13.293311 24761 heartbeater.cc:347] Connected to a master server at 127.21.201.126:34651
I20260812 00:58:13.293654 24761 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:13.294744 24761 heartbeater.cc:511] Master 127.21.201.126:34651 requested a full tablet report, sending...
I20260812 00:58:13.296626 24176 ts_manager.cc:194] Registered new tserver with Master: a6d1a209f49341ee827da80c645e69db (127.21.201.66:40765)
I20260812 00:58:13.297468 24176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.66@KRBTEST.COM'} at 127.21.201.66:40005
I20260812 00:58:13.300583 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 24531
I20260812 00:58:13.300881 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance
I20260812 00:58:13.301503 22309 external_mini_cluster.cc:1428] Setting key e2465347bc43b7185aa715f73665df8a
WARNING: no policy specified for kudu/127.21.201.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
I20260812 00:58:13.364568 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:0
--local_ip_for_outbound_sockets=127.21.201.67
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:34651
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:58:13.671627 24524 heartbeater.cc:503] Master 127.21.201.126:34651 was elected leader, sending a full tablet report...
W20260812 00:58:13.746448 24767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:13.746937 24767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:13.747068 24767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:13.756886 24767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:13.757072 24767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:13.757215 24767 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:58:13.757408 24767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 00:58:13.769146 24767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.201.126:34651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:13.770849 24767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:13.772415 24767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:13.788607 24780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:13.788540 24784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:13.789422 24779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:13.790180 24767 server_base.cc:1034] running on GCE node
I20260812 00:58:13.791152 24767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:13.792537 24767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:13.794016 24767 hybrid_clock.cc:648] HybridClock initialized: now 1786496293793875 us; error 63 us; skew 500 ppm
I20260812 00:58:13.794600 24767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:13 dist-test-slave-7f01 krb5kdc[24098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496293, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:13.800319 24767 init.cc:377] Logged in from keytab as kudu/127.21.201.67@KRBTEST.COM (short username kudu)
I20260812 00:58:13.802680 24767 webserver.cc:460] Webserver started at http://127.21.201.67:38437/ using document root <none> and password file <none>
I20260812 00:58:13.803707 24767 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:13.803932 24767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:13.804392 24767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:13.808552 24767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-2/data/instance:
uuid: "3f3bab6f179b4a02bfb0f60502ebca83"
format_stamp: "Formatted at 2026-08-12 00:58:13 on dist-test-slave-7f01"
server_key: "b6359bbd47734619df26ee5a2334bf5e"
server_key_iv: "2a1fae3067c6b0c5978322cd0c066924"
server_key_version: "encryptionkey@0"
I20260812 00:58:13.809660 24767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance:
uuid: "3f3bab6f179b4a02bfb0f60502ebca83"
format_stamp: "Formatted at 2026-08-12 00:58:13 on dist-test-slave-7f01"
server_key: "b6359bbd47734619df26ee5a2334bf5e"
server_key_iv: "2a1fae3067c6b0c5978322cd0c066924"
server_key_version: "encryptionkey@0"
I20260812 00:58:13.817255 24767 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260812 00:58:13.822708 24796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:13.825043 24767 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260812 00:58:13.825350 24767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "3f3bab6f179b4a02bfb0f60502ebca83"
format_stamp: "Formatted at 2026-08-12 00:58:13 on dist-test-slave-7f01"
server_key: "b6359bbd47734619df26ee5a2334bf5e"
server_key_iv: "2a1fae3067c6b0c5978322cd0c066924"
server_key_version: "encryptionkey@0"
I20260812 00:58:13.825690 24767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:13.857420 24767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:13.864444 24767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:13.864825 24767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:13.869235 24767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:13.869412 24767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:13.869650 24767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:13.869763 24767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 00:58:13 dist-test-slave-7f01 krb5kdc[24098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496293, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:13.904284 24810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:13.870769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:46343 (local address 127.21.201.126:34651)
0812 00:58:13.871421 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:13.871428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:13.873689 (+ 2261us) server_negotiation.cc:407] Connection header received
0812 00:58:13.877584 (+ 3895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:13.877592 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:13.877721 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:13.877833 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:13.881334 (+ 3501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:13.883318 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:13.887869 (+ 4551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:13.889163 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:13.889359 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:13.895605 (+ 6246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:13.895612 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:13.895615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:13.895659 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:13.898892 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:13.899893 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:13.899905 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:13.899907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:13.900058 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:13.900639 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:13.900649 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:13.900655 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:13.901190 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:13.901420 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:13.903174 (+ 1754us) server_negotiation.cc:299] Negotiation successful
0812 00:58:13.903575 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":354,"threads_started":1}
I20260812 00:58:13.906947 24813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:13.872409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:34651 (local address 127.21.201.67:46343)
0812 00:58:13.873466 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:13.873528 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:13.877060 (+ 3532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:13.878637 (+ 1577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:13.878651 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:13.879816 (+ 1165us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:13.881002 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:13.881041 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:13.883577 (+ 2536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:13.883587 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:13.887452 (+ 3865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:13.887498 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:13.889565 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:13.889579 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:13.890082 (+ 503us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:13.891255 (+ 1173us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:13.891304 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:13.895229 (+ 3925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:13.899118 (+ 3889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:13.899131 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:13.899155 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:13.899706 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:13.900188 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:13.900193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:13.900200 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:13.900436 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:13.901506 (+ 1070us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:13.901517 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:13.902307 (+ 790us) client_negotiation.cc:769] Sending connection context
0812 00:58:13.902898 (+ 591us) client_negotiation.cc:240] Negotiation successful
0812 00:58:13.904689 (+ 1791us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":450,"threads_started":1}
I20260812 00:58:13.941932 24767 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:43261
I20260812 00:58:13.941994 25018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:43261 every 8 connection(s)
I20260812 00:58:13.943945 24767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 00:58:13.944511 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 24767
I20260812 00:58:13.944747 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance
I20260812 00:58:13.945413 22309 external_mini_cluster.cc:1428] Setting key 9c1fb1976d596c33f50cc470091e9574
I20260812 00:58:13.946624 24810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:13.923417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:46833 (local address 127.21.201.126:34651)
0812 00:58:13.923717 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:13.923727 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:13.924495 (+ 768us) server_negotiation.cc:407] Connection header received
0812 00:58:13.924771 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:13.924780 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:13.924904 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:13.925091 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:13.927990 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:13.929411 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:13.932578 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:13.933319 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:13.933453 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:13.937694 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:13.937707 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:13.937716 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:13.937799 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:13.942340 (+ 4541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:13.943578 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:13.943585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:13.943588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:13.943913 (+ 325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:13.944513 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:13.944518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:13.944521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:13.944976 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:13.945136 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:13.945835 (+ 699us) server_negotiation.cc:299] Negotiation successful
0812 00:58:13.946175 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 00:58:13.947854 24950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:13.923365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:34651 (local address 127.21.201.67:46833)
0812 00:58:13.924315 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:13.924337 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:13.924574 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:13.925365 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:13.925373 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:13.926846 (+ 1473us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:13.927615 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:13.927646 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:13.929659 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:13.929666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:13.932299 (+ 2633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:13.932328 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:13.933585 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:13.933596 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:13.933911 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:13.935195 (+ 1284us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:13.935234 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:13.937222 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:13.942566 (+ 5344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:13.942577 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:13.942583 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:13.943312 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:13.944141 (+ 829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:13.944147 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:13.944152 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:13.944356 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:13.945234 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:13.945239 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:13.945499 (+ 260us) client_negotiation.cc:769] Sending connection context
0812 00:58:13.946864 (+ 1365us) client_negotiation.cc:240] Negotiation successful
0812 00:58:13.947323 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":599,"threads_started":1}
I20260812 00:58:13.948944 25021 heartbeater.cc:347] Connected to a master server at 127.21.201.126:34651
I20260812 00:58:13.949589 25021 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:13.950769 25021 heartbeater.cc:511] Master 127.21.201.126:34651 requested a full tablet report, sending...
I20260812 00:58:13.952875 24176 ts_manager.cc:194] Registered new tserver with Master: 3f3bab6f179b4a02bfb0f60502ebca83 (127.21.201.67:43261)
I20260812 00:58:13.954151 24176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.67@KRBTEST.COM'} at 127.21.201.67:46343
I20260812 00:58:13.964871 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 00:58:13 dist-test-slave-7f01 krb5kdc[24098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496293, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 00:58:14 dist-test-slave-7f01 krb5kdc[24098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496293, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:14.035916 24810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:14.016359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58444 (local address 127.21.201.126:34651)
0812 00:58:14.016549 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:14.016554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:14.016848 (+ 294us) server_negotiation.cc:407] Connection header received
0812 00:58:14.017038 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:14.017043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:14.017120 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:14.017223 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:14.018765 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.019864 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.021594 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.022178 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.022364 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:14.028031 (+ 5667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:14.028039 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:14.028044 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:14.028109 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:14.032133 (+ 4024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:14.033332 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:14.033337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:14.033340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:14.033497 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:14.033886 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:14.033889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:14.033892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:14.034215 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:14.034537 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:14.035257 (+ 720us) server_negotiation.cc:299] Negotiation successful
0812 00:58:14.035465 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 00:58:14.041538 24176 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:58:14.050233 24176 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:58:14.091311 25048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:14.067875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58712 (local address 127.21.201.65:36177)
0812 00:58:14.077210 (+ 9335us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:14.077224 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:14.079261 (+ 2037us) server_negotiation.cc:407] Connection header received
0812 00:58:14.079522 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:14.079533 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:14.079918 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:14.080277 (+ 359us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:58:14.081533 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.083571 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.087763 (+ 4192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.089457 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.089590 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:14.089806 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:14.090750 (+ 944us) server_negotiation.cc:299] Negotiation successful
0812 00:58:14.090965 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9119,"thread_start_us":8896,"threads_started":1}
I20260812 00:58:14.091367 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:14.067021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:36177 (local address 127.0.0.1:58712)
0812 00:58:14.079017 (+ 11996us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:14.079037 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:14.079187 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:14.080573 (+ 1386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:14.080579 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:14.080609 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:58:14.081288 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:14.081303 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.083857 (+ 2554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.083864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:14.087522 (+ 3658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:14.087540 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.090122 (+ 2582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.090127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:14.090415 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:14.090477 (+ 62us) client_negotiation.cc:769] Sending connection context
0812 00:58:14.090651 (+ 174us) client_negotiation.cc:240] Negotiation successful
0812 00:58:14.090899 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11862,"mutex_wait_us":61,"thread_start_us":10292,"threads_started":1}
I20260812 00:58:14.094146 25045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:14.066221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:40765 (local address 127.0.0.1:52940)
0812 00:58:14.077773 (+ 11552us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:14.077833 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:14.078074 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:14.079566 (+ 1492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:14.079578 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:14.079662 (+ 84us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:58:14.080337 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:14.080357 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.083857 (+ 3500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.083864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:14.090158 (+ 6294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:14.090181 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.092630 (+ 2449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.092636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:14.092796 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:14.093434 (+ 638us) client_negotiation.cc:769] Sending connection context
0812 00:58:14.093532 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 00:58:14.093691 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11282,"mutex_wait_us":3519,"thread_start_us":13076,"threads_started":1}
I20260812 00:58:14.094210 25044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:14.066636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52940 (local address 127.21.201.66:40765)
0812 00:58:14.078692 (+ 12056us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:14.078704 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:14.078741 (+ 37us) server_negotiation.cc:407] Connection header received
0812 00:58:14.078868 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:14.078880 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:14.079200 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:14.079437 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:58:14.081759 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.083625 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.090431 (+ 6806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.092071 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.092233 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:14.092475 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:14.093655 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0812 00:58:14.093874 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11872,"thread_start_us":11723,"threads_started":1}
I20260812 00:58:14.095082 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:14.080197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:43261 (local address 127.0.0.1:42994)
0812 00:58:14.084667 (+ 4470us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:14.084699 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:14.084862 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:14.086716 (+ 1854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:14.086723 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:14.086795 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:58:14.087388 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:14.087408 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.089430 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.089436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:14.092031 (+ 2595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:14.092047 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.094234 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.094243 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:14.094436 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:14.094470 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 00:58:14.094606 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 00:58:14.094755 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4236,"thread_start_us":4001,"threads_started":1}
I20260812 00:58:14.095208 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:14.080249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42994 (local address 127.21.201.67:43261)
0812 00:58:14.085754 (+ 5505us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:14.085767 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:14.085813 (+ 46us) server_negotiation.cc:407] Connection header received
0812 00:58:14.085983 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:14.085996 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:14.086394 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:14.086676 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:58:14.087769 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.089268 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.092251 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.093714 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.093842 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:14.094067 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:14.094656 (+ 589us) server_negotiation.cc:299] Negotiation successful
0812 00:58:14.094834 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5242,"thread_start_us":4989,"threads_started":1}
I20260812 00:58:14.096280 24407 tablet_service.cc:1467] Processing CreateTablet for tablet bbd6a30c11c84186b30eefe0fe269620 (DEFAULT_TABLE table=test-table [id=f465f1b64bd4465d951482a77c9d2c43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:14.098117 24407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbd6a30c11c84186b30eefe0fe269620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:14.098475 24647 tablet_service.cc:1467] Processing CreateTablet for tablet bbd6a30c11c84186b30eefe0fe269620 (DEFAULT_TABLE table=test-table [id=f465f1b64bd4465d951482a77c9d2c43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:14.098881 24882 tablet_service.cc:1467] Processing CreateTablet for tablet bbd6a30c11c84186b30eefe0fe269620 (DEFAULT_TABLE table=test-table [id=f465f1b64bd4465d951482a77c9d2c43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:14.100057 24647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbd6a30c11c84186b30eefe0fe269620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:14.100276 24882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbd6a30c11c84186b30eefe0fe269620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:14.115993 25053 tablet_bootstrap.cc:492] T bbd6a30c11c84186b30eefe0fe269620 P 35285f43f5584e589f0f1f36b5e04aee: Bootstrap starting.
I20260812 00:58:14.121023 25051 tablet_bootstrap.cc:492] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83: Bootstrap starting.
I20260812 00:58:14.122290 25053 tablet_bootstrap.cc:654] T bbd6a30c11c84186b30eefe0fe269620 P 35285f43f5584e589f0f1f36b5e04aee: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:14.124450 25053 log.cc:824] T bbd6a30c11c84186b30eefe0fe269620 P 35285f43f5584e589f0f1f36b5e04aee: Log is configured to *not* fsync() on all Append() calls
I20260812 00:58:14.125998 25055 tablet_bootstrap.cc:492] T bbd6a30c11c84186b30eefe0fe269620 P a6d1a209f49341ee827da80c645e69db: Bootstrap starting.
I20260812 00:58:14.127322 25051 tablet_bootstrap.cc:654] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:14.128615 25051 log.cc:824] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83: Log is configured to *not* fsync() on all Append() calls
I20260812 00:58:14.128746 25053 tablet_bootstrap.cc:492] T bbd6a30c11c84186b30eefe0fe269620 P 35285f43f5584e589f0f1f36b5e04aee: No bootstrap required, opened a new log
I20260812 00:58:14.129071 25053 ts_tablet_manager.cc:1397] T bbd6a30c11c84186b30eefe0fe269620 P 35285f43f5584e589f0f1f36b5e04aee: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260812 00:58:14.130224 25055 tablet_bootstrap.cc:654] T bbd6a30c11c84186b30eefe0fe269620 P a6d1a209f49341ee827da80c645e69db: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:14.131953 25055 log.cc:824] T bbd6a30c11c84186b30eefe0fe269620 P a6d1a209f49341ee827da80c645e69db: Log is configured to *not* fsync() on all Append() calls
I20260812 00:58:14.136083 25051 tablet_bootstrap.cc:492] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83: No bootstrap required, opened a new log
I20260812 00:58:14.136425 25051 ts_tablet_manager.cc:1397] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.008s
I20260812 00:58:14.142648 25055 tablet_bootstrap.cc:492] T bbd6a30c11c84186b30eefe0fe269620 P a6d1a209f49341ee827da80c645e69db: No bootstrap required, opened a new log
I20260812 00:58:14.142589 25053 raft_consensus.cc:348] T bbd6a30c11c84186b30eefe0fe269620 P 35285f43f5584e589f0f1f36b5e04aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:14.142995 25055 ts_tablet_manager.cc:1397] T bbd6a30c11c84186b30eefe0fe269620 P a6d1a209f49341ee827da80c645e69db: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.000s
I20260812 00:58:14.143136 25053 raft_consensus.cc:374] T bbd6a30c11c84186b30eefe0fe269620 P 35285f43f5584e589f0f1f36b5e04aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:14.143301 25053 raft_consensus.cc:729] T bbd6a30c11c84186b30eefe0fe269620 P 35285f43f5584e589f0f1f36b5e04aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35285f43f5584e589f0f1f36b5e04aee, State: Initialized, Role: FOLLOWER
I20260812 00:58:14.144090 25053 consensus_queue.cc:260] T bbd6a30c11c84186b30eefe0fe269620 P 35285f43f5584e589f0f1f36b5e04aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:14.145759 25051 raft_consensus.cc:348] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:14.146306 25053 ts_tablet_manager.cc:1428] T bbd6a30c11c84186b30eefe0fe269620 P 35285f43f5584e589f0f1f36b5e04aee: Time spent starting tablet: real 0.017s user 0.014s sys 0.003s
I20260812 00:58:14.146520 25051 raft_consensus.cc:374] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:14.146644 25051 raft_consensus.cc:729] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3bab6f179b4a02bfb0f60502ebca83, State: Initialized, Role: FOLLOWER
I20260812 00:58:14.147954 25051 consensus_queue.cc:260] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:14.149302 25021 heartbeater.cc:503] Master 127.21.201.126:34651 was elected leader, sending a full tablet report...
I20260812 00:58:14.150132 25051 ts_tablet_manager.cc:1428] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260812 00:58:14.155946 25055 raft_consensus.cc:348] T bbd6a30c11c84186b30eefe0fe269620 P a6d1a209f49341ee827da80c645e69db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:14.156504 25055 raft_consensus.cc:374] T bbd6a30c11c84186b30eefe0fe269620 P a6d1a209f49341ee827da80c645e69db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:14.156617 25055 raft_consensus.cc:729] T bbd6a30c11c84186b30eefe0fe269620 P a6d1a209f49341ee827da80c645e69db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6d1a209f49341ee827da80c645e69db, State: Initialized, Role: FOLLOWER
I20260812 00:58:14.157351 25055 consensus_queue.cc:260] T bbd6a30c11c84186b30eefe0fe269620 P a6d1a209f49341ee827da80c645e69db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:14.158603 24761 heartbeater.cc:503] Master 127.21.201.126:34651 was elected leader, sending a full tablet report...
I20260812 00:58:14.159382 25055 ts_tablet_manager.cc:1428] T bbd6a30c11c84186b30eefe0fe269620 P a6d1a209f49341ee827da80c645e69db: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
W20260812 00:58:14.160544 24525 tablet.cc:2369] T bbd6a30c11c84186b30eefe0fe269620 P 35285f43f5584e589f0f1f36b5e04aee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:58:14.198886 25024 tablet.cc:2369] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:58:14.241236 25062 raft_consensus.cc:482] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:58:14.241940 25062 raft_consensus.cc:504] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:14.244444 25062 leader_election.cc:288] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6d1a209f49341ee827da80c645e69db (127.21.201.66:40765), 35285f43f5584e589f0f1f36b5e04aee (127.21.201.65:36177)
I20260812 00:58:14.254056 25044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:14.245376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:47725 (local address 127.21.201.66:40765)
0812 00:58:14.245685 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:14.245695 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:14.245730 (+ 35us) server_negotiation.cc:407] Connection header received
0812 00:58:14.245976 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:14.245982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:14.246082 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:14.246191 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:58:14.247017 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.248214 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.251148 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.252718 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.252958 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:14.253056 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:14.253610 (+ 554us) server_negotiation.cc:299] Negotiation successful
0812 00:58:14.253734 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260812 00:58:14.254180 24813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:14.245050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:40765 (local address 127.21.201.67:47725)
0812 00:58:14.245559 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:14.245586 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:14.245768 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:14.246253 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:14.246260 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:14.246295 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:58:14.246747 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:14.246772 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.248430 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.248435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:14.250859 (+ 2424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:14.250884 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.253027 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.253034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:14.253373 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:14.253429 (+ 56us) client_negotiation.cc:769] Sending connection context
0812 00:58:14.253569 (+ 140us) client_negotiation.cc:240] Negotiation successful
0812 00:58:14.253749 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":4,"spinlock_wait_cycles":13696}
I20260812 00:58:14.254298 25048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:14.245414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:48595 (local address 127.21.201.65:36177)
0812 00:58:14.245676 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:14.245684 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:14.246336 (+ 652us) server_negotiation.cc:407] Connection header received
0812 00:58:14.246749 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:14.246755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:14.246959 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:14.247140 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:58:14.248093 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.248959 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.251801 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.253131 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.253264 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:14.253316 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:14.253868 (+ 552us) server_negotiation.cc:299] Negotiation successful
0812 00:58:14.254024 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260812 00:58:14.254386 25069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:14.245050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:36177 (local address 127.21.201.67:48595)
0812 00:58:14.246198 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:14.246217 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:14.246344 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:14.247201 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:14.247205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:14.247229 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:58:14.247668 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:14.247678 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.249133 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.249138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:14.251435 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:14.251449 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.253478 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.253498 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:14.253686 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:14.253716 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 00:58:14.253813 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 00:58:14.253962 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1017,"mutex_wait_us":165,"thread_start_us":540,"threads_started":1}
I20260812 00:58:14.255604 24680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd6a30c11c84186b30eefe0fe269620" candidate_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6d1a209f49341ee827da80c645e69db" is_pre_election: true
I20260812 00:58:14.255601 24442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd6a30c11c84186b30eefe0fe269620" candidate_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35285f43f5584e589f0f1f36b5e04aee" is_pre_election: true
I20260812 00:58:14.256259 24680 raft_consensus.cc:2454] T bbd6a30c11c84186b30eefe0fe269620 P a6d1a209f49341ee827da80c645e69db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f3bab6f179b4a02bfb0f60502ebca83 in term 0.
I20260812 00:58:14.256258 24442 raft_consensus.cc:2454] T bbd6a30c11c84186b30eefe0fe269620 P 35285f43f5584e589f0f1f36b5e04aee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f3bab6f179b4a02bfb0f60502ebca83 in term 0.
I20260812 00:58:14.257118 24802 leader_election.cc:302] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 35285f43f5584e589f0f1f36b5e04aee, 3f3bab6f179b4a02bfb0f60502ebca83, a6d1a209f49341ee827da80c645e69db; no voters:
I20260812 00:58:14.257649 25062 raft_consensus.cc:2781] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:58:14.257822 25062 raft_consensus.cc:482] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:58:14.257951 25062 raft_consensus.cc:3037] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:14.260778 25062 raft_consensus.cc:504] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:14.261618 25062 leader_election.cc:288] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83 [CANDIDATE]: Term 1 election: Requested vote from peers a6d1a209f49341ee827da80c645e69db (127.21.201.66:40765), 35285f43f5584e589f0f1f36b5e04aee (127.21.201.65:36177)
I20260812 00:58:14.262271 24680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd6a30c11c84186b30eefe0fe269620" candidate_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6d1a209f49341ee827da80c645e69db"
I20260812 00:58:14.262531 24680 raft_consensus.cc:3037] T bbd6a30c11c84186b30eefe0fe269620 P a6d1a209f49341ee827da80c645e69db [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:14.262604 24442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd6a30c11c84186b30eefe0fe269620" candidate_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35285f43f5584e589f0f1f36b5e04aee"
I20260812 00:58:14.262998 24442 raft_consensus.cc:3037] T bbd6a30c11c84186b30eefe0fe269620 P 35285f43f5584e589f0f1f36b5e04aee [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:14.265337 24680 raft_consensus.cc:2454] T bbd6a30c11c84186b30eefe0fe269620 P a6d1a209f49341ee827da80c645e69db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f3bab6f179b4a02bfb0f60502ebca83 in term 1.
I20260812 00:58:14.266012 24803 leader_election.cc:302] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f3bab6f179b4a02bfb0f60502ebca83, a6d1a209f49341ee827da80c645e69db; no voters:
I20260812 00:58:14.266175 24442 raft_consensus.cc:2454] T bbd6a30c11c84186b30eefe0fe269620 P 35285f43f5584e589f0f1f36b5e04aee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f3bab6f179b4a02bfb0f60502ebca83 in term 1.
I20260812 00:58:14.266489 25062 raft_consensus.cc:2781] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:58:14.267300 25062 raft_consensus.cc:686] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83 [term 1 LEADER]: Becoming Leader. State: Replica: 3f3bab6f179b4a02bfb0f60502ebca83, State: Running, Role: LEADER
I20260812 00:58:14.268054 25062 consensus_queue.cc:237] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:14.274591 24176 catalog_manager.cc:5495] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f3bab6f179b4a02bfb0f60502ebca83 (127.21.201.67). New cstate: current_term: 1 leader_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } health_report { overall_health: HEALTHY } } }
W20260812 00:58:14.293330 24762 tablet.cc:2369] T bbd6a30c11c84186b30eefe0fe269620 P a6d1a209f49341ee827da80c645e69db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:58:14.352705 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:14.344211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43004 (local address 127.21.201.67:43261)
0812 00:58:14.344553 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:14.344561 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:14.344761 (+ 200us) server_negotiation.cc:407] Connection header received
0812 00:58:14.345290 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:14.345296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:14.345417 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:14.345636 (+ 219us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:58:14.346175 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.347183 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.349307 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.349844 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.350029 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:14.351004 (+ 975us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:58:14.351808 (+ 804us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:58:14.351942 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:14.352225 (+ 283us) server_negotiation.cc:299] Negotiation successful
0812 00:58:14.352321 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260812 00:58:14.364682 24680 raft_consensus.cc:1261] T bbd6a30c11c84186b30eefe0fe269620 P a6d1a209f49341ee827da80c645e69db [term 1 FOLLOWER]: Refusing update from remote peer 3f3bab6f179b4a02bfb0f60502ebca83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:58:14.364754 24442 raft_consensus.cc:1261] T bbd6a30c11c84186b30eefe0fe269620 P 35285f43f5584e589f0f1f36b5e04aee [term 1 FOLLOWER]: Refusing update from remote peer 3f3bab6f179b4a02bfb0f60502ebca83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:58:14.365980 25062 consensus_queue.cc:1035] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:58:14.366631 25070 consensus_queue.cc:1035] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:58:14.385324 25078 mvcc.cc:205] Tried to move back new op lower bound from 7317488821702819840 to 7317488821328322560. Current Snapshot: MvccSnapshot[applied={T|T < 7317488821702819840}]
I20260812 00:58:14.411631 24171 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:58:14.412371 24171 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:58:14.418700 24171 catalog_manager.cc:5782] T 00000000000000000000000000000000 P caf58d56f64846f6a93c0e452d73e974: Sending DeleteTablet for 3 replicas of tablet bbd6a30c11c84186b30eefe0fe269620
I20260812 00:58:14.420617 24407 tablet_service.cc:1514] Processing DeleteTablet for tablet bbd6a30c11c84186b30eefe0fe269620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:58:14 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:58712
I20260812 00:58:14.420948 24882 tablet_service.cc:1514] Processing DeleteTablet for tablet bbd6a30c11c84186b30eefe0fe269620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:58:14 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:42994
I20260812 00:58:14.422116 24647 tablet_service.cc:1514] Processing DeleteTablet for tablet bbd6a30c11c84186b30eefe0fe269620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:58:14 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:52940
I20260812 00:58:14.423583 25097 tablet_replica.cc:330] T bbd6a30c11c84186b30eefe0fe269620 P a6d1a209f49341ee827da80c645e69db: stopping tablet replica
I20260812 00:58:14.424321 25097 raft_consensus.cc:2229] T bbd6a30c11c84186b30eefe0fe269620 P a6d1a209f49341ee827da80c645e69db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:58:14.425040 25097 raft_consensus.cc:2258] T bbd6a30c11c84186b30eefe0fe269620 P a6d1a209f49341ee827da80c645e69db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:58:14.426344 24171 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 00:58:14.427928 25097 ts_tablet_manager.cc:1905] T bbd6a30c11c84186b30eefe0fe269620 P a6d1a209f49341ee827da80c645e69db: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 00:58:14.428910 24171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:58:14.432144 25095 tablet_replica.cc:330] T bbd6a30c11c84186b30eefe0fe269620 P 35285f43f5584e589f0f1f36b5e04aee: stopping tablet replica
I20260812 00:58:14.432888 25095 raft_consensus.cc:2229] T bbd6a30c11c84186b30eefe0fe269620 P 35285f43f5584e589f0f1f36b5e04aee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:58:14.433545 25095 raft_consensus.cc:2258] T bbd6a30c11c84186b30eefe0fe269620 P 35285f43f5584e589f0f1f36b5e04aee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:58:14.434544 25094 tablet_replica.cc:330] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83: stopping tablet replica
I20260812 00:58:14.435103 25094 raft_consensus.cc:2229] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:58:14.436184 25094 raft_consensus.cc:2258] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:58:14.436533 25095 ts_tablet_manager.cc:1905] T bbd6a30c11c84186b30eefe0fe269620 P 35285f43f5584e589f0f1f36b5e04aee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:58:14.439584 25097 ts_tablet_manager.cc:1918] T bbd6a30c11c84186b30eefe0fe269620 P a6d1a209f49341ee827da80c645e69db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:58:14.442045 25094 ts_tablet_manager.cc:1905] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:58:14.443707 24407 tablet_service.cc:1467] Processing CreateTablet for tablet 306856cf3cc748acab0dd810909bed8f (DEFAULT_TABLE table=test-table [id=a9a85227ab4d483eb83d7b3c29332632]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:14.444283 24407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 306856cf3cc748acab0dd810909bed8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:14.444880 25095 ts_tablet_manager.cc:1918] T bbd6a30c11c84186b30eefe0fe269620 P 35285f43f5584e589f0f1f36b5e04aee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:58:14.445081 25095 log.cc:1196] T bbd6a30c11c84186b30eefe0fe269620 P 35285f43f5584e589f0f1f36b5e04aee: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-0/wal/wals/bbd6a30c11c84186b30eefe0fe269620
I20260812 00:58:14.445963 25095 ts_tablet_manager.cc:1939] T bbd6a30c11c84186b30eefe0fe269620 P 35285f43f5584e589f0f1f36b5e04aee: Deleting consensus metadata
I20260812 00:58:14.448503 24151 catalog_manager.cc:4841] TS 35285f43f5584e589f0f1f36b5e04aee (127.21.201.65:36177): tablet bbd6a30c11c84186b30eefe0fe269620 (table test-table [id=f465f1b64bd4465d951482a77c9d2c43]) successfully deleted
I20260812 00:58:14.442646 24647 tablet_service.cc:1467] Processing CreateTablet for tablet 306856cf3cc748acab0dd810909bed8f (DEFAULT_TABLE table=test-table [id=a9a85227ab4d483eb83d7b3c29332632]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:14.448758 24882 tablet_service.cc:1467] Processing CreateTablet for tablet 306856cf3cc748acab0dd810909bed8f (DEFAULT_TABLE table=test-table [id=a9a85227ab4d483eb83d7b3c29332632]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:14.449275 24882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 306856cf3cc748acab0dd810909bed8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:14.449278 24647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 306856cf3cc748acab0dd810909bed8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:14.452185 25097 log.cc:1196] T bbd6a30c11c84186b30eefe0fe269620 P a6d1a209f49341ee827da80c645e69db: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/bbd6a30c11c84186b30eefe0fe269620
I20260812 00:58:14.452901 25097 ts_tablet_manager.cc:1939] T bbd6a30c11c84186b30eefe0fe269620 P a6d1a209f49341ee827da80c645e69db: Deleting consensus metadata
I20260812 00:58:14.455263 24154 catalog_manager.cc:4841] TS a6d1a209f49341ee827da80c645e69db (127.21.201.66:40765): tablet bbd6a30c11c84186b30eefe0fe269620 (table test-table [id=f465f1b64bd4465d951482a77c9d2c43]) successfully deleted
I20260812 00:58:14.451877 25094 ts_tablet_manager.cc:1918] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:58:14.456053 25094 log.cc:1196] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/bbd6a30c11c84186b30eefe0fe269620
I20260812 00:58:14.457139 25055 tablet_bootstrap.cc:492] T 306856cf3cc748acab0dd810909bed8f P a6d1a209f49341ee827da80c645e69db: Bootstrap starting.
I20260812 00:58:14.457854 25053 tablet_bootstrap.cc:492] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee: Bootstrap starting.
I20260812 00:58:14.458443 25094 ts_tablet_manager.cc:1939] T bbd6a30c11c84186b30eefe0fe269620 P 3f3bab6f179b4a02bfb0f60502ebca83: Deleting consensus metadata
I20260812 00:58:14.459128 25055 tablet_bootstrap.cc:654] T 306856cf3cc748acab0dd810909bed8f P a6d1a209f49341ee827da80c645e69db: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:14.459767 25053 tablet_bootstrap.cc:654] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:14.461751 24154 catalog_manager.cc:4841] TS 3f3bab6f179b4a02bfb0f60502ebca83 (127.21.201.67:43261): tablet bbd6a30c11c84186b30eefe0fe269620 (table test-table [id=f465f1b64bd4465d951482a77c9d2c43]) successfully deleted
I20260812 00:58:14.470625 25051 tablet_bootstrap.cc:492] T 306856cf3cc748acab0dd810909bed8f P 3f3bab6f179b4a02bfb0f60502ebca83: Bootstrap starting.
I20260812 00:58:14.472195 25053 tablet_bootstrap.cc:492] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee: No bootstrap required, opened a new log
I20260812 00:58:14.472477 25055 tablet_bootstrap.cc:492] T 306856cf3cc748acab0dd810909bed8f P a6d1a209f49341ee827da80c645e69db: No bootstrap required, opened a new log
I20260812 00:58:14.472492 25053 ts_tablet_manager.cc:1397] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.007s
I20260812 00:58:14.472704 25055 ts_tablet_manager.cc:1397] T 306856cf3cc748acab0dd810909bed8f P a6d1a209f49341ee827da80c645e69db: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20260812 00:58:14.473152 25051 tablet_bootstrap.cc:654] T 306856cf3cc748acab0dd810909bed8f P 3f3bab6f179b4a02bfb0f60502ebca83: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:14.473450 25053 raft_consensus.cc:348] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:14.473708 25053 raft_consensus.cc:374] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:14.473803 25053 raft_consensus.cc:729] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35285f43f5584e589f0f1f36b5e04aee, State: Initialized, Role: FOLLOWER
I20260812 00:58:14.473851 25055 raft_consensus.cc:348] T 306856cf3cc748acab0dd810909bed8f P a6d1a209f49341ee827da80c645e69db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:14.474125 25055 raft_consensus.cc:374] T 306856cf3cc748acab0dd810909bed8f P a6d1a209f49341ee827da80c645e69db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:14.474221 25055 raft_consensus.cc:729] T 306856cf3cc748acab0dd810909bed8f P a6d1a209f49341ee827da80c645e69db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6d1a209f49341ee827da80c645e69db, State: Initialized, Role: FOLLOWER
I20260812 00:58:14.474135 25053 consensus_queue.cc:260] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:14.474529 25055 consensus_queue.cc:260] T 306856cf3cc748acab0dd810909bed8f P a6d1a209f49341ee827da80c645e69db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:14.475296 25055 ts_tablet_manager.cc:1428] T 306856cf3cc748acab0dd810909bed8f P a6d1a209f49341ee827da80c645e69db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:58:14.477669 25053 ts_tablet_manager.cc:1428] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260812 00:58:14.481978 25051 tablet_bootstrap.cc:492] T 306856cf3cc748acab0dd810909bed8f P 3f3bab6f179b4a02bfb0f60502ebca83: No bootstrap required, opened a new log
I20260812 00:58:14.482589 25051 ts_tablet_manager.cc:1397] T 306856cf3cc748acab0dd810909bed8f P 3f3bab6f179b4a02bfb0f60502ebca83: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260812 00:58:14.483841 25051 raft_consensus.cc:348] T 306856cf3cc748acab0dd810909bed8f P 3f3bab6f179b4a02bfb0f60502ebca83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:14.484087 25051 raft_consensus.cc:374] T 306856cf3cc748acab0dd810909bed8f P 3f3bab6f179b4a02bfb0f60502ebca83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:14.484264 25051 raft_consensus.cc:729] T 306856cf3cc748acab0dd810909bed8f P 3f3bab6f179b4a02bfb0f60502ebca83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3bab6f179b4a02bfb0f60502ebca83, State: Initialized, Role: FOLLOWER
I20260812 00:58:14.484635 25051 consensus_queue.cc:260] T 306856cf3cc748acab0dd810909bed8f P 3f3bab6f179b4a02bfb0f60502ebca83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:14.485301 25051 ts_tablet_manager.cc:1428] T 306856cf3cc748acab0dd810909bed8f P 3f3bab6f179b4a02bfb0f60502ebca83: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260812 00:58:14.583721 25060 raft_consensus.cc:482] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:58:14.584267 25060 raft_consensus.cc:504] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:14.587049 25060 leader_election.cc:288] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6d1a209f49341ee827da80c645e69db (127.21.201.66:40765), 3f3bab6f179b4a02bfb0f60502ebca83 (127.21.201.67:43261)
I20260812 00:58:14.597313 25044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:14.588261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:41231 (local address 127.21.201.66:40765)
0812 00:58:14.588699 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:14.588708 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:14.589668 (+ 960us) server_negotiation.cc:407] Connection header received
0812 00:58:14.590038 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:14.590045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:14.590132 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:14.590204 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:58:14.591310 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.592589 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.594919 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.595940 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.596176 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:14.596298 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:14.596759 (+ 461us) server_negotiation.cc:299] Negotiation successful
0812 00:58:14.596922 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260812 00:58:14.597481 25104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:14.587623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:40765 (local address 127.21.201.65:41231)
0812 00:58:14.589503 (+ 1880us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:14.589529 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:14.589810 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:14.590438 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:14.590442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:14.590471 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:58:14.591012 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:14.591045 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.592750 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.592757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:14.594721 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:14.594734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.596351 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.596359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:14.596517 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:14.596548 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 00:58:14.596678 (+ 130us) client_negotiation.cc:240] Negotiation successful
0812 00:58:14.596989 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1706,"mutex_wait_us":116,"thread_start_us":1423,"threads_started":1}
I20260812 00:58:14.598515 24680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "306856cf3cc748acab0dd810909bed8f" candidate_uuid: "35285f43f5584e589f0f1f36b5e04aee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6d1a209f49341ee827da80c645e69db" is_pre_election: true
I20260812 00:58:14.598834 24680 raft_consensus.cc:2454] T 306856cf3cc748acab0dd810909bed8f P a6d1a209f49341ee827da80c645e69db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35285f43f5584e589f0f1f36b5e04aee in term 0.
I20260812 00:58:14.599436 24331 leader_election.cc:302] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35285f43f5584e589f0f1f36b5e04aee, a6d1a209f49341ee827da80c645e69db; no voters:
I20260812 00:58:14.600026 25060 raft_consensus.cc:2781] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:58:14.600193 25060 raft_consensus.cc:482] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:58:14.600284 25060 raft_consensus.cc:3037] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:14.601894 25106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:14.589904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:43261 (local address 127.21.201.65:51391)
0812 00:58:14.592432 (+ 2528us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:14.592450 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:14.592622 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:14.593093 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:14.593098 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:14.593125 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:58:14.593577 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:14.593589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.595994 (+ 2405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.596000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:14.598759 (+ 2759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:14.598779 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.600869 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.600876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:14.601081 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:14.601116 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 00:58:14.601225 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 00:58:14.601396 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2392,"mutex_wait_us":21,"thread_start_us":1325,"threads_started":1}
I20260812 00:58:14.602489 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:14.590459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:51391 (local address 127.21.201.67:43261)
0812 00:58:14.590698 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:14.590706 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:14.592558 (+ 1852us) server_negotiation.cc:407] Connection header received
0812 00:58:14.592834 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:14.592841 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:14.592937 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:14.593047 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:58:14.594046 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.595788 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.599026 (+ 3238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.600591 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.600804 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:14.601764 (+ 960us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:14.601833 (+ 69us) server_negotiation.cc:299] Negotiation successful
0812 00:58:14.602040 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260812 00:58:14.603915 24931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "306856cf3cc748acab0dd810909bed8f" candidate_uuid: "35285f43f5584e589f0f1f36b5e04aee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" is_pre_election: true
I20260812 00:58:14.604580 24931 raft_consensus.cc:2454] T 306856cf3cc748acab0dd810909bed8f P 3f3bab6f179b4a02bfb0f60502ebca83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35285f43f5584e589f0f1f36b5e04aee in term 0.
I20260812 00:58:14.681268 25060 raft_consensus.cc:504] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:14.682889 25060 leader_election.cc:288] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee [CANDIDATE]: Term 1 election: Requested vote from peers a6d1a209f49341ee827da80c645e69db (127.21.201.66:40765), 3f3bab6f179b4a02bfb0f60502ebca83 (127.21.201.67:43261)
I20260812 00:58:14.683396 24680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "306856cf3cc748acab0dd810909bed8f" candidate_uuid: "35285f43f5584e589f0f1f36b5e04aee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6d1a209f49341ee827da80c645e69db"
I20260812 00:58:14.683838 24931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "306856cf3cc748acab0dd810909bed8f" candidate_uuid: "35285f43f5584e589f0f1f36b5e04aee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f3bab6f179b4a02bfb0f60502ebca83"
I20260812 00:58:14.683972 24680 raft_consensus.cc:3037] T 306856cf3cc748acab0dd810909bed8f P a6d1a209f49341ee827da80c645e69db [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:14.684137 24931 raft_consensus.cc:3037] T 306856cf3cc748acab0dd810909bed8f P 3f3bab6f179b4a02bfb0f60502ebca83 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:14.688580 24680 raft_consensus.cc:2454] T 306856cf3cc748acab0dd810909bed8f P a6d1a209f49341ee827da80c645e69db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35285f43f5584e589f0f1f36b5e04aee in term 1.
I20260812 00:58:14.689671 24331 leader_election.cc:302] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35285f43f5584e589f0f1f36b5e04aee, a6d1a209f49341ee827da80c645e69db; no voters:
I20260812 00:58:14.690277 25060 raft_consensus.cc:2781] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:58:14.691448 25060 raft_consensus.cc:686] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee [term 1 LEADER]: Becoming Leader. State: Replica: 35285f43f5584e589f0f1f36b5e04aee, State: Running, Role: LEADER
I20260812 00:58:14.692094 25060 consensus_queue.cc:237] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:14.697429 24176 catalog_manager.cc:5495] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee reported cstate change: term changed from 0 to 1, leader changed from <none> to 35285f43f5584e589f0f1f36b5e04aee (127.21.201.65). New cstate: current_term: 1 leader_uuid: "35285f43f5584e589f0f1f36b5e04aee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } health_report { overall_health: UNKNOWN } } }
Aug 12 00:58:14 dist-test-slave-7f01 krb5kdc[24098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496291, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:14.766568 25126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:14.746481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:34651 (local address 127.0.0.1:58458)
0812 00:58:14.747301 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:14.747326 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:14.747604 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:14.748254 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:14.748260 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:14.749573 (+ 1313us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:14.749917 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:14.749931 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.751321 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.751327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:14.753012 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:14.753045 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.754131 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.754139 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:14.754346 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:14.755301 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:14.755326 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:14.758898 (+ 3572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:14.763463 (+ 4565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:14.763485 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:14.763495 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:14.764117 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:14.764563 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:14.764567 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:14.764570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:14.764712 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:14.765306 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:14.765315 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:14.765490 (+ 175us) client_negotiation.cc:769] Sending connection context
0812 00:58:14.765675 (+ 185us) client_negotiation.cc:240] Negotiation successful
0812 00:58:14.766063 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":5,"thread_start_us":401,"threads_started":1}
I20260812 00:58:14.766706 25122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:14.746517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58458 (local address 127.21.201.126:34651)
0812 00:58:14.747938 (+ 1421us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:14.747943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:14.747966 (+ 23us) server_negotiation.cc:407] Connection header received
0812 00:58:14.748014 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:14.748017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:14.748073 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:14.748169 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:14.750182 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.751152 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.753244 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:14.753720 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:14.753827 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:14.759352 (+ 5525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:14.759359 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:14.759363 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:14.759419 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:14.763229 (+ 3810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:14.764256 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:14.764272 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:14.764275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:14.764424 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:14.764840 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:14.764852 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:14.764854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:14.765155 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:14.765337 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:14.765770 (+ 433us) server_negotiation.cc:299] Negotiation successful
0812 00:58:14.766194 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":196,"server-negotiator.queue_time_us":1288,"thread_start_us":1094,"threads_started":1}
I20260812 00:58:14.774781 24176 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:58458:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 00:58:14.776536 24176 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:58:14.786935 24407 tablet_service.cc:1467] Processing CreateTablet for tablet 45e8f84b613f41158e2c5a58dd1cdeaa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b69dee0f9d9143c0889d79c086496405]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 00:58:14.787834 24407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e8f84b613f41158e2c5a58dd1cdeaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:14.788014 24647 tablet_service.cc:1467] Processing CreateTablet for tablet 45e8f84b613f41158e2c5a58dd1cdeaa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b69dee0f9d9143c0889d79c086496405]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 00:58:14.788818 24647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e8f84b613f41158e2c5a58dd1cdeaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:14.790956 24931 raft_consensus.cc:2454] T 306856cf3cc748acab0dd810909bed8f P 3f3bab6f179b4a02bfb0f60502ebca83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35285f43f5584e589f0f1f36b5e04aee in term 1.
I20260812 00:58:14.792665 24882 tablet_service.cc:1467] Processing CreateTablet for tablet 45e8f84b613f41158e2c5a58dd1cdeaa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b69dee0f9d9143c0889d79c086496405]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 00:58:14.793499 24882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e8f84b613f41158e2c5a58dd1cdeaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:14.799396 25055 tablet_bootstrap.cc:492] T 45e8f84b613f41158e2c5a58dd1cdeaa P a6d1a209f49341ee827da80c645e69db: Bootstrap starting.
I20260812 00:58:14.801190 25055 tablet_bootstrap.cc:654] T 45e8f84b613f41158e2c5a58dd1cdeaa P a6d1a209f49341ee827da80c645e69db: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:14.802064 25053 tablet_bootstrap.cc:492] T 45e8f84b613f41158e2c5a58dd1cdeaa P 35285f43f5584e589f0f1f36b5e04aee: Bootstrap starting.
I20260812 00:58:14.802104 25051 tablet_bootstrap.cc:492] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83: Bootstrap starting.
I20260812 00:58:14.804184 25053 tablet_bootstrap.cc:654] T 45e8f84b613f41158e2c5a58dd1cdeaa P 35285f43f5584e589f0f1f36b5e04aee: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:14.804688 25051 tablet_bootstrap.cc:654] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:14.814913 25053 tablet_bootstrap.cc:492] T 45e8f84b613f41158e2c5a58dd1cdeaa P 35285f43f5584e589f0f1f36b5e04aee: No bootstrap required, opened a new log
I20260812 00:58:14.815276 25053 ts_tablet_manager.cc:1397] T 45e8f84b613f41158e2c5a58dd1cdeaa P 35285f43f5584e589f0f1f36b5e04aee: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260812 00:58:14.816562 25053 raft_consensus.cc:348] T 45e8f84b613f41158e2c5a58dd1cdeaa P 35285f43f5584e589f0f1f36b5e04aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:14.816987 25053 raft_consensus.cc:374] T 45e8f84b613f41158e2c5a58dd1cdeaa P 35285f43f5584e589f0f1f36b5e04aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:14.817118 25053 raft_consensus.cc:729] T 45e8f84b613f41158e2c5a58dd1cdeaa P 35285f43f5584e589f0f1f36b5e04aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35285f43f5584e589f0f1f36b5e04aee, State: Initialized, Role: FOLLOWER
I20260812 00:58:14.817163 25055 tablet_bootstrap.cc:492] T 45e8f84b613f41158e2c5a58dd1cdeaa P a6d1a209f49341ee827da80c645e69db: No bootstrap required, opened a new log
I20260812 00:58:14.817397 25055 ts_tablet_manager.cc:1397] T 45e8f84b613f41158e2c5a58dd1cdeaa P a6d1a209f49341ee827da80c645e69db: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.003s
I20260812 00:58:14.817596 25053 consensus_queue.cc:260] T 45e8f84b613f41158e2c5a58dd1cdeaa P 35285f43f5584e589f0f1f36b5e04aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:14.818341 25055 raft_consensus.cc:348] T 45e8f84b613f41158e2c5a58dd1cdeaa P a6d1a209f49341ee827da80c645e69db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:14.818429 25060 tablet_replica.cc:439] T 45e8f84b613f41158e2c5a58dd1cdeaa P 35285f43f5584e589f0f1f36b5e04aee: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } } }
I20260812 00:58:14.818647 25055 raft_consensus.cc:374] T 45e8f84b613f41158e2c5a58dd1cdeaa P a6d1a209f49341ee827da80c645e69db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:14.818737 25055 raft_consensus.cc:729] T 45e8f84b613f41158e2c5a58dd1cdeaa P a6d1a209f49341ee827da80c645e69db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6d1a209f49341ee827da80c645e69db, State: Initialized, Role: FOLLOWER
I20260812 00:58:14.818783 25060 tablet_replica.cc:442] T 45e8f84b613f41158e2c5a58dd1cdeaa P 35285f43f5584e589f0f1f36b5e04aee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:58:14.818814 25051 tablet_bootstrap.cc:492] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83: No bootstrap required, opened a new log
I20260812 00:58:14.819145 25051 ts_tablet_manager.cc:1397] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.008s
I20260812 00:58:14.819085 25055 consensus_queue.cc:260] T 45e8f84b613f41158e2c5a58dd1cdeaa P a6d1a209f49341ee827da80c645e69db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:14.819664 25066 tablet_replica.cc:439] T 45e8f84b613f41158e2c5a58dd1cdeaa P a6d1a209f49341ee827da80c645e69db: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } } }
I20260812 00:58:14.820071 25066 tablet_replica.cc:442] T 45e8f84b613f41158e2c5a58dd1cdeaa P a6d1a209f49341ee827da80c645e69db: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:58:14.821899 25055 ts_tablet_manager.cc:1428] T 45e8f84b613f41158e2c5a58dd1cdeaa P a6d1a209f49341ee827da80c645e69db: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260812 00:58:14.822860 25051 raft_consensus.cc:348] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:14.823108 25053 ts_tablet_manager.cc:1428] T 45e8f84b613f41158e2c5a58dd1cdeaa P 35285f43f5584e589f0f1f36b5e04aee: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260812 00:58:14.823303 25051 raft_consensus.cc:374] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:14.823443 25051 raft_consensus.cc:729] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3bab6f179b4a02bfb0f60502ebca83, State: Initialized, Role: FOLLOWER
I20260812 00:58:14.824059 25051 consensus_queue.cc:260] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:14.824746 25091 tablet_replica.cc:439] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } } }
I20260812 00:58:14.825379 25091 tablet_replica.cc:442] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:58:14.826941 25051 ts_tablet_manager.cc:1428] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20260812 00:58:14.893332 25091 raft_consensus.cc:482] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:58:14.893668 25091 raft_consensus.cc:504] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:14.894714 25091 leader_election.cc:288] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35285f43f5584e589f0f1f36b5e04aee (127.21.201.65:36177), a6d1a209f49341ee827da80c645e69db (127.21.201.66:40765)
I20260812 00:58:14.895447 24680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e8f84b613f41158e2c5a58dd1cdeaa" candidate_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6d1a209f49341ee827da80c645e69db" is_pre_election: true
I20260812 00:58:14.895524 24442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e8f84b613f41158e2c5a58dd1cdeaa" candidate_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35285f43f5584e589f0f1f36b5e04aee" is_pre_election: true
I20260812 00:58:14.895820 24680 raft_consensus.cc:2454] T 45e8f84b613f41158e2c5a58dd1cdeaa P a6d1a209f49341ee827da80c645e69db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f3bab6f179b4a02bfb0f60502ebca83 in term 0.
I20260812 00:58:14.895901 24442 raft_consensus.cc:2454] T 45e8f84b613f41158e2c5a58dd1cdeaa P 35285f43f5584e589f0f1f36b5e04aee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f3bab6f179b4a02bfb0f60502ebca83 in term 0.
I20260812 00:58:14.896394 24803 leader_election.cc:302] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f3bab6f179b4a02bfb0f60502ebca83, a6d1a209f49341ee827da80c645e69db; no voters:
I20260812 00:58:14.896837 25091 raft_consensus.cc:2781] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:58:14.897017 25091 raft_consensus.cc:482] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:58:14.897145 25091 raft_consensus.cc:3037] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:14.990106 25091 raft_consensus.cc:504] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:14.991163 25091 leader_election.cc:288] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83 [CANDIDATE]: Term 1 election: Requested vote from peers 35285f43f5584e589f0f1f36b5e04aee (127.21.201.65:36177), a6d1a209f49341ee827da80c645e69db (127.21.201.66:40765)
I20260812 00:58:14.992329 24442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e8f84b613f41158e2c5a58dd1cdeaa" candidate_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35285f43f5584e589f0f1f36b5e04aee"
I20260812 00:58:14.992832 24442 raft_consensus.cc:3037] T 45e8f84b613f41158e2c5a58dd1cdeaa P 35285f43f5584e589f0f1f36b5e04aee [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:14.993839 24680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e8f84b613f41158e2c5a58dd1cdeaa" candidate_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6d1a209f49341ee827da80c645e69db"
I20260812 00:58:14.994179 24680 raft_consensus.cc:3037] T 45e8f84b613f41158e2c5a58dd1cdeaa P a6d1a209f49341ee827da80c645e69db [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:15.085628 24680 raft_consensus.cc:2454] T 45e8f84b613f41158e2c5a58dd1cdeaa P a6d1a209f49341ee827da80c645e69db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f3bab6f179b4a02bfb0f60502ebca83 in term 1.
I20260812 00:58:15.085695 24442 raft_consensus.cc:2454] T 45e8f84b613f41158e2c5a58dd1cdeaa P 35285f43f5584e589f0f1f36b5e04aee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f3bab6f179b4a02bfb0f60502ebca83 in term 1.
I20260812 00:58:15.086637 24803 leader_election.cc:302] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 35285f43f5584e589f0f1f36b5e04aee, 3f3bab6f179b4a02bfb0f60502ebca83, a6d1a209f49341ee827da80c645e69db; no voters:
I20260812 00:58:15.087217 25091 raft_consensus.cc:2781] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:58:15.087868 25091 raft_consensus.cc:686] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83 [term 1 LEADER]: Becoming Leader. State: Replica: 3f3bab6f179b4a02bfb0f60502ebca83, State: Running, Role: LEADER
I20260812 00:58:15.088225 25091 consensus_queue.cc:237] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } }
I20260812 00:58:15.092589 25138 tablet_replica.cc:439] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83: TxnStatusTablet state changed. Reason: New leader 3f3bab6f179b4a02bfb0f60502ebca83. Latest consensus state: current_term: 1 leader_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } } }
I20260812 00:58:15.093075 25138 tablet_replica.cc:442] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:58:15.094115 25143 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 00:58:15.094523 25143 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 00:58:15.094504 24170 catalog_manager.cc:5495] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f3bab6f179b4a02bfb0f60502ebca83 (127.21.201.67). New cstate: current_term: 1 leader_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } health_report { overall_health: HEALTHY } } }
I20260812 00:58:15.127418 25145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:15.120457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43016 (local address 127.21.201.67:43261)
0812 00:58:15.121232 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:15.121237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:15.121254 (+ 17us) server_negotiation.cc:407] Connection header received
0812 00:58:15.121330 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:15.121333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:15.121410 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:15.121572 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:58:15.122025 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:15.122945 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:15.124940 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:15.125505 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:15.125684 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:15.126336 (+ 652us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:58:15.126486 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:58:15.126604 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:15.126955 (+ 351us) server_negotiation.cc:299] Negotiation successful
0812 00:58:15.127052 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":386,"threads_started":1}
I20260812 00:58:15.127504 25126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:15.120387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:43261 (local address 127.0.0.1:43016)
0812 00:58:15.120682 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:15.120708 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:15.120907 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:15.121605 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:15.121610 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:15.121623 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:58:15.121831 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:15.121840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:15.123120 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:15.123127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:15.124603 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:15.124704 (+ 101us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:15.125717 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:15.125724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:15.125984 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:15.126076 (+ 92us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:58:15.126679 (+ 603us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:58:15.126710 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 00:58:15.126860 (+ 150us) client_negotiation.cc:240] Negotiation successful
0812 00:58:15.127071 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260812 00:58:15.134064 24680 raft_consensus.cc:1261] T 45e8f84b613f41158e2c5a58dd1cdeaa P a6d1a209f49341ee827da80c645e69db [term 1 FOLLOWER]: Refusing update from remote peer 3f3bab6f179b4a02bfb0f60502ebca83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:58:15.134023 24442 raft_consensus.cc:1261] T 45e8f84b613f41158e2c5a58dd1cdeaa P 35285f43f5584e589f0f1f36b5e04aee [term 1 FOLLOWER]: Refusing update from remote peer 3f3bab6f179b4a02bfb0f60502ebca83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:58:15.134841 25091 consensus_queue.cc:1035] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:58:15.135251 25138 consensus_queue.cc:1035] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:58:15.147214 25066 tablet_replica.cc:439] T 45e8f84b613f41158e2c5a58dd1cdeaa P a6d1a209f49341ee827da80c645e69db: TxnStatusTablet state changed. Reason: New leader 3f3bab6f179b4a02bfb0f60502ebca83. Latest consensus state: current_term: 1 leader_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } } }
I20260812 00:58:15.147614 25066 tablet_replica.cc:442] T 45e8f84b613f41158e2c5a58dd1cdeaa P a6d1a209f49341ee827da80c645e69db: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:58:15.151589 25060 tablet_replica.cc:439] T 45e8f84b613f41158e2c5a58dd1cdeaa P 35285f43f5584e589f0f1f36b5e04aee: TxnStatusTablet state changed. Reason: New leader 3f3bab6f179b4a02bfb0f60502ebca83. Latest consensus state: current_term: 1 leader_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } } }
I20260812 00:58:15.151935 25060 tablet_replica.cc:442] T 45e8f84b613f41158e2c5a58dd1cdeaa P 35285f43f5584e589f0f1f36b5e04aee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:58:15.153230 25060 consensus_queue.cc:1035] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:58:15.154924 25149 mvcc.cc:205] Tried to move back new op lower bound from 7317488824859033600 to 7317488824684494848. Current Snapshot: MvccSnapshot[applied={T|T < 7317488824859033600}]
I20260812 00:58:15.164611 25060 tablet_replica.cc:439] T 45e8f84b613f41158e2c5a58dd1cdeaa P 35285f43f5584e589f0f1f36b5e04aee: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } } }
I20260812 00:58:15.165103 25060 tablet_replica.cc:442] T 45e8f84b613f41158e2c5a58dd1cdeaa P 35285f43f5584e589f0f1f36b5e04aee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:58:15.180514 25153 mvcc.cc:205] Tried to move back new op lower bound from 7317488824859033600 to 7317488824684494848. Current Snapshot: MvccSnapshot[applied={T|T < 7317488824859033600}]
I20260812 00:58:15.180676 25091 tablet_replica.cc:439] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } } }
I20260812 00:58:15.180998 25091 tablet_replica.cc:442] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:58:15.181490 25091 tablet_replica.cc:439] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } } }
I20260812 00:58:15.182183 25091 tablet_replica.cc:442] T 45e8f84b613f41158e2c5a58dd1cdeaa P 3f3bab6f179b4a02bfb0f60502ebca83: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:58:15.186357 25066 tablet_replica.cc:439] T 45e8f84b613f41158e2c5a58dd1cdeaa P a6d1a209f49341ee827da80c645e69db: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35285f43f5584e589f0f1f36b5e04aee" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 36177 } } peers { permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 } } peers { permanent_uuid: "3f3bab6f179b4a02bfb0f60502ebca83" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 43261 } } }
I20260812 00:58:15.186704 25066 tablet_replica.cc:442] T 45e8f84b613f41158e2c5a58dd1cdeaa P a6d1a209f49341ee827da80c645e69db: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:58:15.190608 25107 consensus_queue.cc:1035] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6d1a209f49341ee827da80c645e69db" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:58:15.215822 25173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:15.198609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58722 (local address 127.21.201.65:36177)
0812 00:58:15.205122 (+ 6513us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:15.205133 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:15.206085 (+ 952us) server_negotiation.cc:407] Connection header received
0812 00:58:15.206318 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:15.206326 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:15.206498 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:15.206853 (+ 355us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:58:15.207456 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:15.209038 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:15.211211 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:15.212337 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:15.212547 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:15.213332 (+ 785us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:58:15.214488 (+ 1156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:58:15.214744 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:15.214938 (+ 194us) server_negotiation.cc:299] Negotiation successful
0812 00:58:15.215076 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6245,"thread_start_us":6017,"threads_started":1}
I20260812 00:58:15.240952 25145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:15.231607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:35645 (local address 127.21.201.67:43261)
0812 00:58:15.231997 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:15.232005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:15.233040 (+ 1035us) server_negotiation.cc:407] Connection header received
0812 00:58:15.233344 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:15.233351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:15.233462 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:15.233594 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:58:15.234639 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:15.236438 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:15.238125 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:15.238958 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:15.239140 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:15.239644 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:58:15.239879 (+ 235us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:58:15.239998 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:15.240408 (+ 410us) server_negotiation.cc:299] Negotiation successful
0812 00:58:15.240542 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260812 00:58:15.241098 25178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:15.231614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:43261 (local address 127.21.201.65:35645)
0812 00:58:15.232866 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:15.232885 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:15.233156 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:15.233774 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:15.233782 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:15.233805 (+ 23us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:58:15.234425 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:15.234450 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:15.236616 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:15.236621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:15.237941 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:15.237957 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:15.239190 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:15.239195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:15.239338 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:15.239431 (+ 93us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:58:15.240150 (+ 719us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:58:15.240178 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 00:58:15.240346 (+ 168us) client_negotiation.cc:240] Negotiation successful
0812 00:58:15.240547 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"thread_start_us":671,"threads_started":1}
I20260812 00:58:15.320212 25173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:15.300412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:39895 (local address 127.21.201.65:36177)
0812 00:58:15.300710 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:15.300718 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:15.312862 (+ 12144us) server_negotiation.cc:407] Connection header received
0812 00:58:15.313247 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:15.313254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:15.313366 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:15.313443 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:58:15.314584 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:15.315425 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:15.317660 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:15.318318 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:15.318435 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:15.319224 (+ 789us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:58:15.319375 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:58:15.319446 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:15.319714 (+ 268us) server_negotiation.cc:299] Negotiation successful
0812 00:58:15.319850 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 00:58:15.320358 25183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:15.300720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:36177 (local address 127.21.201.67:39895)
0812 00:58:15.312662 (+ 11942us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:15.312686 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:15.312970 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:15.313736 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:15.313741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:15.313754 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:58:15.314369 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:15.314382 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:15.315593 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:15.315599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:15.317456 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:15.317474 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:15.318677 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:15.318687 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:15.318866 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:15.318973 (+ 107us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:58:15.319560 (+ 587us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:58:15.319569 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 00:58:15.319683 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 00:58:15.319870 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11653,"thread_start_us":11452,"threads_started":1}
I20260812 00:58:15.380703 24170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:58:15.381116 24170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:58:15.385095 24170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P caf58d56f64846f6a93c0e452d73e974: Sending DeleteTablet for 3 replicas of tablet 306856cf3cc748acab0dd810909bed8f
I20260812 00:58:15.386171 24407 tablet_service.cc:1514] Processing DeleteTablet for tablet 306856cf3cc748acab0dd810909bed8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:58:15 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:58712
I20260812 00:58:15.386546 24647 tablet_service.cc:1514] Processing DeleteTablet for tablet 306856cf3cc748acab0dd810909bed8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:58:15 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:52940
I20260812 00:58:15.387413 24882 tablet_service.cc:1514] Processing DeleteTablet for tablet 306856cf3cc748acab0dd810909bed8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:58:15 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:42994
I20260812 00:58:15.393826 25186 tablet_replica.cc:330] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee: stopping tablet replica
I20260812 00:58:15.394616 25186 raft_consensus.cc:2229] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:58:15.395330 25186 raft_consensus.cc:2258] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:58:15.400755 25191 tablet_replica.cc:330] T 306856cf3cc748acab0dd810909bed8f P 3f3bab6f179b4a02bfb0f60502ebca83: stopping tablet replica
I20260812 00:58:15.401528 25187 tablet_replica.cc:330] T 306856cf3cc748acab0dd810909bed8f P a6d1a209f49341ee827da80c645e69db: stopping tablet replica
I20260812 00:58:15.402371 25191 raft_consensus.cc:2229] T 306856cf3cc748acab0dd810909bed8f P 3f3bab6f179b4a02bfb0f60502ebca83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:58:15.402858 25191 raft_consensus.cc:2258] T 306856cf3cc748acab0dd810909bed8f P 3f3bab6f179b4a02bfb0f60502ebca83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:58:15.403227 25187 raft_consensus.cc:2229] T 306856cf3cc748acab0dd810909bed8f P a6d1a209f49341ee827da80c645e69db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:58:15.403841 25187 raft_consensus.cc:2258] T 306856cf3cc748acab0dd810909bed8f P a6d1a209f49341ee827da80c645e69db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:58:15.405238 25191 ts_tablet_manager.cc:1905] T 306856cf3cc748acab0dd810909bed8f P 3f3bab6f179b4a02bfb0f60502ebca83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:58:15.406810 25187 ts_tablet_manager.cc:1905] T 306856cf3cc748acab0dd810909bed8f P a6d1a209f49341ee827da80c645e69db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:58:15.408772 25186 ts_tablet_manager.cc:1905] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 12 00:58:15 dist-test-slave-7f01 krb5kdc[24098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496293, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.201.65@KRBTEST.COM
I20260812 00:58:15.441859 25173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:15.401849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58728 (local address 127.21.201.65:36177)
0812 00:58:15.402106 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:15.402112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:15.417784 (+ 15672us) server_negotiation.cc:407] Connection header received
0812 00:58:15.418011 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:15.418019 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:15.418131 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:15.418301 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:15.420766 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:15.422035 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:15.424202 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:15.425010 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:15.425147 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:15.431783 (+ 6636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:15.431811 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:15.431846 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:15.431926 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:15.437387 (+ 5461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:15.438665 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:15.438683 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:15.438687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:15.438903 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:15.439474 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:15.439480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:15.439485 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:15.440172 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:15.440618 (+ 446us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:15.440968 (+ 350us) server_negotiation.cc:299] Negotiation successful
0812 00:58:15.441281 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260812 00:58:15.443384 24515 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58728
I20260812 00:58:15.471206 25198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:15.447051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58474 (local address 127.21.201.126:34651)
0812 00:58:15.448053 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:15.448061 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:15.451038 (+ 2977us) server_negotiation.cc:407] Connection header received
0812 00:58:15.451388 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:15.451396 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:15.451508 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:15.451652 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:15.453898 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:15.455342 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:15.457629 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:15.458453 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:15.458582 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:15.462942 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:15.462952 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:15.462958 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:15.463022 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:15.467387 (+ 4365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:15.468645 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:15.468652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:15.468656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:15.468842 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:15.469331 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:15.469338 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:15.469341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:15.469771 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:15.470070 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:15.470462 (+ 392us) server_negotiation.cc:299] Negotiation successful
0812 00:58:15.470754 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":559,"threads_started":1}
W20260812 00:58:15.471982 24170 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58474
I20260812 00:58:15.474496 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 24296
I20260812 00:58:15.495206 25186 ts_tablet_manager.cc:1918] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:58:15.495715 25186 log.cc:1196] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-0/wal/wals/306856cf3cc748acab0dd810909bed8f
I20260812 00:58:15.496502 25186 ts_tablet_manager.cc:1939] T 306856cf3cc748acab0dd810909bed8f P 35285f43f5584e589f0f1f36b5e04aee: Deleting consensus metadata
I20260812 00:58:15.512264 25173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:15.476075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.21.201.65:36177)
0812 00:58:15.476498 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:15.476506 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:15.480485 (+ 3979us) server_negotiation.cc:407] Connection header received
0812 00:58:15.480676 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:15.480683 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:15.480821 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:15.480996 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:15.482961 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:15.484258 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:15.486681 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:15.487669 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:15.487927 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:15.501810 (+ 13883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:15.501821 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:15.501888 (+ 67us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:15.501968 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:15.507332 (+ 5364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:15.508806 (+ 1474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:15.508813 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:15.508818 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:15.509070 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:15.509708 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:15.509713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:15.509716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:15.510392 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:15.511055 (+ 663us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:15.511364 (+ 309us) server_negotiation.cc:299] Negotiation successful
0812 00:58:15.511776 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252}
W20260812 00:58:15.513633 24515 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58730
W20260812 00:58:15.514441 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 24296: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 00:58:15.514680 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 24296
I20260812 00:58:15.591068 25191 ts_tablet_manager.cc:1918] T 306856cf3cc748acab0dd810909bed8f P 3f3bab6f179b4a02bfb0f60502ebca83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:58:15.591239 25187 ts_tablet_manager.cc:1918] T 306856cf3cc748acab0dd810909bed8f P a6d1a209f49341ee827da80c645e69db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:58:15.591446 25187 log.cc:1196] T 306856cf3cc748acab0dd810909bed8f P a6d1a209f49341ee827da80c645e69db: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/306856cf3cc748acab0dd810909bed8f
I20260812 00:58:15.591574 25191 log.cc:1196] T 306856cf3cc748acab0dd810909bed8f P 3f3bab6f179b4a02bfb0f60502ebca83: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/306856cf3cc748acab0dd810909bed8f
I20260812 00:58:15.592067 25187 ts_tablet_manager.cc:1939] T 306856cf3cc748acab0dd810909bed8f P a6d1a209f49341ee827da80c645e69db: Deleting consensus metadata
I20260812 00:58:15.592346 25191 ts_tablet_manager.cc:1939] T 306856cf3cc748acab0dd810909bed8f P 3f3bab6f179b4a02bfb0f60502ebca83: Deleting consensus metadata
I20260812 00:58:15.597088 24154 catalog_manager.cc:4841] TS a6d1a209f49341ee827da80c645e69db (127.21.201.66:40765): tablet 306856cf3cc748acab0dd810909bed8f (table test-table [id=a9a85227ab4d483eb83d7b3c29332632]) successfully deleted
I20260812 00:58:15.599105 24154 catalog_manager.cc:4841] TS 3f3bab6f179b4a02bfb0f60502ebca83 (127.21.201.67:43261): tablet 306856cf3cc748acab0dd810909bed8f (table test-table [id=a9a85227ab4d483eb83d7b3c29332632]) successfully deleted
W20260812 00:58:15.603348 24151 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.201.65:36177 (error 108)
I20260812 00:58:15.603726 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 24531
I20260812 00:58:15.607416 25207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:15.605549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:36177 (local address 127.0.0.1:58736)
0812 00:58:15.606907 (+ 1358us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:15.607130 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1203,"thread_start_us":1057,"threads_started":1}
W20260812 00:58:15.607808 24151 catalog_manager.cc:4584] TS 35285f43f5584e589f0f1f36b5e04aee (127.21.201.65:36177): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 306856cf3cc748acab0dd810909bed8f: Network error: Client connection negotiation failed: client connection to 127.21.201.65:36177: connect: Connection refused (error 111)
Aug 12 00:58:15 dist-test-slave-7f01 krb5kdc[24098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496293, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.201.66@KRBTEST.COM
I20260812 00:58:15.630342 25205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:15.605147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52950 (local address 127.21.201.66:40765)
0812 00:58:15.605786 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:15.605796 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:15.605824 (+ 28us) server_negotiation.cc:407] Connection header received
0812 00:58:15.605917 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:15.605926 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:15.606035 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:15.606210 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:15.608264 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:15.609770 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:15.611710 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:15.612610 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:15.612807 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:15.618846 (+ 6039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:15.618874 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:15.618903 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:15.618969 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:15.625227 (+ 6258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:15.626700 (+ 1473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:15.626714 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:15.626719 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:15.627000 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:15.627644 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:15.627652 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:15.627658 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:15.628429 (+ 771us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:15.628951 (+ 522us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:15.629411 (+ 460us) server_negotiation.cc:299] Negotiation successful
0812 00:58:15.629798 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":277,"threads_started":1}
W20260812 00:58:15.631649 24750 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52950
W20260812 00:58:15.632936 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 24531: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 00:58:15.633221 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 24531
I20260812 00:58:15.662029 25207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:15.661271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:36177 (local address 127.0.0.1:58752)
0812 00:58:15.661586 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:15.661733 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260812 00:58:15.664064 25207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:15.663119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:36177 (local address 127.0.0.1:58756)
0812 00:58:15.663523 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:15.663716 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260812 00:58:15.678006 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 24767
Aug 12 00:58:15 dist-test-slave-7f01 krb5kdc[24098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496293, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.201.67@KRBTEST.COM
I20260812 00:58:15.698963 25145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:15.679084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43032 (local address 127.21.201.67:43261)
0812 00:58:15.679313 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:15.679321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:15.679345 (+ 24us) server_negotiation.cc:407] Connection header received
0812 00:58:15.679441 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:15.679447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:15.679543 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:15.679660 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:15.681890 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:15.683006 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:15.684572 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:15.685042 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:15.685206 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:15.691271 (+ 6065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:15.691294 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:15.691312 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:15.691370 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:15.695398 (+ 4028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:15.696454 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:15.696462 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:15.696464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:15.696633 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:15.697188 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:15.697193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:15.697195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:15.697615 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:15.698037 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:15.698307 (+ 270us) server_negotiation.cc:299] Negotiation successful
0812 00:58:15.698568 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260812 00:58:15.699925 25014 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43032
W20260812 00:58:15.700830 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 24767: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 00:58:15.701128 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 24767
I20260812 00:58:15.725708 25207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:15.724939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:36177 (local address 127.0.0.1:58770)
0812 00:58:15.725324 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:15.725520 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260812 00:58:15.727654 25207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:15.726629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:36177 (local address 127.0.0.1:58782)
0812 00:58:15.726966 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:15.727271 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260812 00:58:15.755153 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 24118
I20260812 00:58:15.756719 24240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:15.925107 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 24118
I20260812 00:58:15.925137 25207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:15.924340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:36177 (local address 127.0.0.1:58786)
0812 00:58:15.924741 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:15.924888 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228}
2026-08-12T00:58:15Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (6346 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:58:16 dist-test-slave-7f01 krb5kdc[25213](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:58:16 dist-test-slave-7f01 krb5kdc[25213](info): set up 2 sockets
Aug 12 00:58:16 dist-test-slave-7f01 krb5kdc[25213](info): commencing operation
krb5kdc: starting...
W20260812 00:58:17.764966 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.778s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 00:58:17 dist-test-slave-7f01 krb5kdc[25213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:58:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:58:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 00:58:18.036695 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:38635
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:38389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:38635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:18.434535 25236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:18.434994 25236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:18.435178 25236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:18.444254 25236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:58:18.444491 25236 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 00:58:18.444612 25236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:18.444741 25236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:18.444842 25236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:58:18.444947 25236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:58:18.457154 25236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:38389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:38635
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:38635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:18.458706 25236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:18.460274 25236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:18.474800 25244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:18.474869 25248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:18.474720 25246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:18.475991 25236 server_base.cc:1034] running on GCE node
I20260812 00:58:18.476989 25236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:18.478821 25236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:18.480216 25236 hybrid_clock.cc:648] HybridClock initialized: now 1786496298480140 us; error 59 us; skew 500 ppm
I20260812 00:58:18.480712 25236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:18 dist-test-slave-7f01 krb5kdc[25213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496298, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:18.489794 25236 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 00:58:18.491905 25236 webserver.cc:460] Webserver started at http://127.21.201.126:40447/ using document root <none> and password file <none>
I20260812 00:58:18.492792 25236 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:18.492908 25236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:18.493294 25236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:18.496848 25236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "aeefe634cbe24087837667b7ba19fe06"
format_stamp: "Formatted at 2026-08-12 00:58:18 on dist-test-slave-7f01"
server_key: "09fd29787a40282dcf4866a1db6aa09d"
server_key_iv: "286bd562c00b6cde6d11bc73032a6449"
server_key_version: "encryptionkey@0"
I20260812 00:58:18.497817 25236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "aeefe634cbe24087837667b7ba19fe06"
format_stamp: "Formatted at 2026-08-12 00:58:18 on dist-test-slave-7f01"
server_key: "09fd29787a40282dcf4866a1db6aa09d"
server_key_iv: "286bd562c00b6cde6d11bc73032a6449"
server_key_version: "encryptionkey@0"
I20260812 00:58:18.505987 25236 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.003s
I20260812 00:58:18.511373 25257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:18.513474 25236 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 00:58:18.513732 25236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "aeefe634cbe24087837667b7ba19fe06"
format_stamp: "Formatted at 2026-08-12 00:58:18 on dist-test-slave-7f01"
server_key: "09fd29787a40282dcf4866a1db6aa09d"
server_key_iv: "286bd562c00b6cde6d11bc73032a6449"
server_key_version: "encryptionkey@0"
I20260812 00:58:18.514053 25236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:18.537124 25236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:18.542346 25236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:18.542699 25236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:18.570899 25236 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:38635
I20260812 00:58:18.570909 25366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:38635 every 8 connection(s)
I20260812 00:58:18.572644 25236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 00:58:18.576866 25368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:18.580772 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 25236
I20260812 00:58:18.581048 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 00:58:18.581962 22309 external_mini_cluster.cc:1428] Setting key 23d70352506a0207e5624c8bf1408ab7
I20260812 00:58:18.592998 25368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06: Bootstrap starting.
I20260812 00:58:18.597618 25368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:18.599298 25368 log.cc:824] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06: Log is configured to *not* fsync() on all Append() calls
Aug 12 00:58:18 dist-test-slave-7f01 krb5kdc[25213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:18.604642 25368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06: No bootstrap required, opened a new log
I20260812 00:58:18.613659 25373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:18.585107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58206 (local address 127.21.201.126:38635)
0812 00:58:18.586091 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:18.586104 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:18.586150 (+ 46us) server_negotiation.cc:407] Connection header received
0812 00:58:18.588114 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:18.588143 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:18.588636 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:18.589037 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:18.591249 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:18.592763 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:18.594707 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:18.595681 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:18.596080 (+ 399us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:18.603170 (+ 7090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:18.603181 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:18.603203 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:18.603256 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:18.607245 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:18.608746 (+ 1501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:18.608758 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:18.608767 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:18.609060 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:18.609803 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:18.609810 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:18.609813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:18.610683 (+ 870us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:18.611007 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:18.611659 (+ 652us) server_negotiation.cc:299] Negotiation successful
0812 00:58:18.612350 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":467,"threads_started":1}
I20260812 00:58:18.615437 25368 raft_consensus.cc:348] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeefe634cbe24087837667b7ba19fe06" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38635 } }
I20260812 00:58:18.616272 25368 raft_consensus.cc:374] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:18.616429 25368 raft_consensus.cc:729] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeefe634cbe24087837667b7ba19fe06, State: Initialized, Role: FOLLOWER
I20260812 00:58:18.617249 25368 consensus_queue.cc:260] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeefe634cbe24087837667b7ba19fe06" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38635 } }
I20260812 00:58:18.617622 25368 raft_consensus.cc:388] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:58:18.617789 25368 raft_consensus.cc:482] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:58:18.618034 25368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:18.620725 25368 raft_consensus.cc:504] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeefe634cbe24087837667b7ba19fe06" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38635 } }
I20260812 00:58:18.621340 25368 leader_election.cc:302] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aeefe634cbe24087837667b7ba19fe06; no voters:
I20260812 00:58:18.622058 25368 leader_election.cc:288] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:58:18.622285 25379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:58:18.623886 25379 raft_consensus.cc:686] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06 [term 1 LEADER]: Becoming Leader. State: Replica: aeefe634cbe24087837667b7ba19fe06, State: Running, Role: LEADER
I20260812 00:58:18.624636 25368 sys_catalog.cc:564] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:58:18.624598 25379 consensus_queue.cc:237] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeefe634cbe24087837667b7ba19fe06" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38635 } }
I20260812 00:58:18.627874 25381 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aeefe634cbe24087837667b7ba19fe06. Latest consensus state: current_term: 1 leader_uuid: "aeefe634cbe24087837667b7ba19fe06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeefe634cbe24087837667b7ba19fe06" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38635 } } }
I20260812 00:58:18.628441 25381 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06 [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:18.628026 25380 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aeefe634cbe24087837667b7ba19fe06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeefe634cbe24087837667b7ba19fe06" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38635 } } }
I20260812 00:58:18.629027 25380 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06 [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:18.634559 25402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:58:18.639577 25402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:58:18.651700 25402 catalog_manager.cc:1347] Generated new cluster ID: 9df1d3d95d3e4eda8a2e1729c661c971
I20260812 00:58:18.651907 25402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:58:18.666235 25402 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:58:18.668604 25402 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:58:18.687463 25402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06: Generated new TSK 0
I20260812 00:58:18.688786 25402 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.201.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
I20260812 00:58:18.764779 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:0
--local_ip_for_outbound_sockets=127.21.201.65
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38635
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:38389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:19.172257 25423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:19.172626 25423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:19.172819 25423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:19.180938 25423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:19.181097 25423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:19.181195 25423 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 00:58:19.181243 25423 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:58:19.181418 25423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 00:58:19.194531 25423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:38389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.201.126:38635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:19.197014 25423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:19.198861 25423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:19.216198 25439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:19.216351 25433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:19.216308 25435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:19.217386 25423 server_base.cc:1034] running on GCE node
I20260812 00:58:19.218152 25423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:19.219511 25423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:19.220921 25423 hybrid_clock.cc:648] HybridClock initialized: now 1786496299220823 us; error 86 us; skew 500 ppm
I20260812 00:58:19.221508 25423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:19 dist-test-slave-7f01 krb5kdc[25213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496299, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:19.226634 25423 init.cc:377] Logged in from keytab as kudu/127.21.201.65@KRBTEST.COM (short username kudu)
I20260812 00:58:19.229133 25423 webserver.cc:460] Webserver started at http://127.21.201.65:43685/ using document root <none> and password file <none>
I20260812 00:58:19.230046 25423 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:19.230232 25423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:19.230611 25423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:19.234141 25423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-0/data/instance:
uuid: "0dbc61bb753e46a9be28f8e789ad8b28"
format_stamp: "Formatted at 2026-08-12 00:58:19 on dist-test-slave-7f01"
server_key: "660b2abd3f921707a98559ae685aa47a"
server_key_iv: "0547526fc3d70de46b6cbdf454c1ae6f"
server_key_version: "encryptionkey@0"
I20260812 00:58:19.235070 25423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance:
uuid: "0dbc61bb753e46a9be28f8e789ad8b28"
format_stamp: "Formatted at 2026-08-12 00:58:19 on dist-test-slave-7f01"
server_key: "660b2abd3f921707a98559ae685aa47a"
server_key_iv: "0547526fc3d70de46b6cbdf454c1ae6f"
server_key_version: "encryptionkey@0"
I20260812 00:58:19.241307 25423 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 00:58:19.245945 25446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:19.248344 25423 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260812 00:58:19.248662 25423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "0dbc61bb753e46a9be28f8e789ad8b28"
format_stamp: "Formatted at 2026-08-12 00:58:19 on dist-test-slave-7f01"
server_key: "660b2abd3f921707a98559ae685aa47a"
server_key_iv: "0547526fc3d70de46b6cbdf454c1ae6f"
server_key_version: "encryptionkey@0"
I20260812 00:58:19.248978 25423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:19.266249 25423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:19.271620 25423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:19.272070 25423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:19.275275 25423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:19.275467 25423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:19.275620 25423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:19.275731 25423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 00:58:19 dist-test-slave-7f01 krb5kdc[25213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496299, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:19.307518 25463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:19.276324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:42167 (local address 127.21.201.126:38635)
0812 00:58:19.276866 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:19.276875 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:19.278288 (+ 1413us) server_negotiation.cc:407] Connection header received
0812 00:58:19.281052 (+ 2764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:19.281062 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:19.281206 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:19.281350 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:19.284044 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:19.285843 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:19.289885 (+ 4042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:19.290782 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:19.291041 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:19.297328 (+ 6287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:19.297341 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:19.297350 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:19.297424 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:19.302137 (+ 4713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:19.303591 (+ 1454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:19.303598 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:19.303604 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:19.303840 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:19.304884 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:19.304888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:19.304891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:19.305224 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:19.305361 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:19.306903 (+ 1542us) server_negotiation.cc:299] Negotiation successful
0812 00:58:19.307220 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":260,"threads_started":1}
I20260812 00:58:19.309885 25466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:19.277094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38635 (local address 127.21.201.65:42167)
0812 00:58:19.278070 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:19.278120 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:19.280559 (+ 2439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:19.281721 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:19.281739 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:19.282665 (+ 926us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:19.283735 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:19.283800 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:19.286323 (+ 2523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:19.286329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:19.289391 (+ 3062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:19.289428 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:19.291218 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:19.291231 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:19.291639 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:19.292989 (+ 1350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:19.293034 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:19.296825 (+ 3791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:19.302397 (+ 5572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:19.302429 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:19.302452 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:19.303331 (+ 879us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:19.304024 (+ 693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:19.304032 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:19.304039 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:19.304693 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:19.305409 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:19.305427 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:19.306165 (+ 738us) client_negotiation.cc:769] Sending connection context
0812 00:58:19.306682 (+ 517us) client_negotiation.cc:240] Negotiation successful
0812 00:58:19.307605 (+ 923us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":443,"threads_started":1}
I20260812 00:58:19.351823 25463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:19.331208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:56239 (local address 127.21.201.126:38635)
0812 00:58:19.331423 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:19.331428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:19.332935 (+ 1507us) server_negotiation.cc:407] Connection header received
0812 00:58:19.333479 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:19.333488 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:19.333609 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:19.333736 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:19.335651 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:19.337193 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:19.339588 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:19.340416 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:19.340592 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:19.343170 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:19.343180 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:19.343188 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:19.343251 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:19.347651 (+ 4400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:19.349393 (+ 1742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:19.349401 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:19.349405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:19.349588 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:19.350055 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:19.350061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:19.350064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:19.350433 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:19.350609 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:19.351078 (+ 469us) server_negotiation.cc:299] Negotiation successful
0812 00:58:19.351377 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 00:58:19.352321 25594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:19.331399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38635 (local address 127.21.201.65:56239)
0812 00:58:19.332748 (+ 1349us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:19.332779 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:19.333080 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:19.333814 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:19.333826 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:19.334677 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:19.335256 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:19.335274 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:19.337610 (+ 2336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:19.337625 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:19.339285 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:19.339310 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:19.340717 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:19.340725 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:19.340930 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:19.341706 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:19.341726 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:19.342844 (+ 1118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:19.348038 (+ 5194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:19.348044 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:19.348048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:19.349161 (+ 1113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:19.349756 (+ 595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:19.349761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:19.349764 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:19.349920 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:19.350580 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:19.350585 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:19.350812 (+ 227us) client_negotiation.cc:769] Sending connection context
0812 00:58:19.350977 (+ 165us) client_negotiation.cc:240] Negotiation successful
0812 00:58:19.351421 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1079,"thread_start_us":689,"threads_started":1}
I20260812 00:58:19.357403 25423 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:44581
I20260812 00:58:19.357414 25664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:44581 every 8 connection(s)
I20260812 00:58:19.359189 25423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 00:58:19.362299 25665 heartbeater.cc:347] Connected to a master server at 127.21.201.126:38635
I20260812 00:58:19.362649 25665 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:19.363552 25665 heartbeater.cc:511] Master 127.21.201.126:38635 requested a full tablet report, sending...
I20260812 00:58:19.366169 25288 ts_manager.cc:194] Registered new tserver with Master: 0dbc61bb753e46a9be28f8e789ad8b28 (127.21.201.65:44581)
I20260812 00:58:19.366204 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 25423
I20260812 00:58:19.366490 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance
I20260812 00:58:19.367286 22309 external_mini_cluster.cc:1428] Setting key 4c21009715b83d2d83af738442708e50
I20260812 00:58:19.368247 25288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.65@KRBTEST.COM'} at 127.21.201.65:42167
WARNING: no policy specified for kudu/127.21.201.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
I20260812 00:58:19.425045 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:0
--local_ip_for_outbound_sockets=127.21.201.66
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38635
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:38389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:19.820472 25672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:19.820818 25672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:19.820940 25672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:19.829538 25672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:19.829716 25672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:19.829869 25672 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 00:58:19.829964 25672 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:58:19.830044 25672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 00:58:19.841862 25672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:38389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.201.126:38635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:19.843384 25672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:19.844888 25672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:19.860605 25680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:19.860798 25682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:19.861431 25679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:19.862661 25672 server_base.cc:1034] running on GCE node
I20260812 00:58:19.863695 25672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:19.865407 25672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:19.866827 25672 hybrid_clock.cc:648] HybridClock initialized: now 1786496299866763 us; error 68 us; skew 500 ppm
I20260812 00:58:19.867341 25672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:19 dist-test-slave-7f01 krb5kdc[25213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496299, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:19.872499 25672 init.cc:377] Logged in from keytab as kudu/127.21.201.66@KRBTEST.COM (short username kudu)
I20260812 00:58:19.874624 25672 webserver.cc:460] Webserver started at http://127.21.201.66:34051/ using document root <none> and password file <none>
I20260812 00:58:19.875522 25672 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:19.875705 25672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:19.876227 25672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:19.881616 25672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-1/data/instance:
uuid: "ad6137c90eb04e17bccecb4fa4977244"
format_stamp: "Formatted at 2026-08-12 00:58:19 on dist-test-slave-7f01"
server_key: "3bbc796063746d16f536a522fc19dd81"
server_key_iv: "2ef12f0270ff7fc41bc57f1e073aa817"
server_key_version: "encryptionkey@0"
I20260812 00:58:19.882901 25672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance:
uuid: "ad6137c90eb04e17bccecb4fa4977244"
format_stamp: "Formatted at 2026-08-12 00:58:19 on dist-test-slave-7f01"
server_key: "3bbc796063746d16f536a522fc19dd81"
server_key_iv: "2ef12f0270ff7fc41bc57f1e073aa817"
server_key_version: "encryptionkey@0"
I20260812 00:58:19.891309 25672 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.005s
I20260812 00:58:19.896067 25695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:19.897836 25672 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 00:58:19.898087 25672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "ad6137c90eb04e17bccecb4fa4977244"
format_stamp: "Formatted at 2026-08-12 00:58:19 on dist-test-slave-7f01"
server_key: "3bbc796063746d16f536a522fc19dd81"
server_key_iv: "2ef12f0270ff7fc41bc57f1e073aa817"
server_key_version: "encryptionkey@0"
I20260812 00:58:19.898373 25672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:19.922690 25672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:19.929693 25672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:19.930116 25672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:19.934103 25672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:19.934255 25672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:19.934396 25672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:19.934486 25672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 00:58:19 dist-test-slave-7f01 krb5kdc[25213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496299, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:19.970534 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:19.935083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:46277 (local address 127.21.201.126:38635)
0812 00:58:19.935893 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:19.935901 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:19.937157 (+ 1256us) server_negotiation.cc:407] Connection header received
0812 00:58:19.939469 (+ 2312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:19.939475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:19.939582 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:19.939721 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:19.942982 (+ 3261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:19.944485 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:19.948592 (+ 4107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:19.949541 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:19.949771 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:19.957658 (+ 7887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:19.957665 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:19.957670 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:19.957731 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:19.962567 (+ 4836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:19.964552 (+ 1985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:19.964563 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:19.964570 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:19.964857 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:19.966223 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:19.966230 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:19.966235 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:19.966911 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:19.967191 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:19.969638 (+ 2447us) server_negotiation.cc:299] Negotiation successful
0812 00:58:19.970058 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":435,"threads_started":1}
I20260812 00:58:19.973091 25710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:19.935884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38635 (local address 127.21.201.66:46277)
0812 00:58:19.936928 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:19.936984 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:19.939103 (+ 2119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:19.940049 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:19.940058 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:19.940992 (+ 934us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:19.942354 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:19.942425 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:19.944976 (+ 2551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:19.944987 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:19.948262 (+ 3275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:19.948296 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:19.949967 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:19.949980 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:19.950459 (+ 479us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:19.952551 (+ 2092us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:19.952611 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:19.957225 (+ 4614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:19.963006 (+ 5781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:19.963031 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:19.963059 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:19.964250 (+ 1191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:19.965298 (+ 1048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:19.965306 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:19.965311 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:19.965820 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:19.967298 (+ 1478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:19.967319 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:19.968756 (+ 1437us) client_negotiation.cc:769] Sending connection context
0812 00:58:19.969452 (+ 696us) client_negotiation.cc:240] Negotiation successful
0812 00:58:19.970288 (+ 836us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":428,"threads_started":1}
I20260812 00:58:20.018460 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:19.989026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:39265 (local address 127.21.201.126:38635)
0812 00:58:19.989754 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:19.989765 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:19.993967 (+ 4202us) server_negotiation.cc:407] Connection header received
0812 00:58:19.994237 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:19.994245 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:19.994356 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:19.994444 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:19.996296 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:19.997562 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.000371 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.001158 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.001274 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:20.008939 (+ 7665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:20.008946 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:20.008950 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:20.008994 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:20.013612 (+ 4618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:20.014805 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:20.014813 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:20.014818 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:20.014998 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:20.015644 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:20.015652 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:20.015658 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:20.016290 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:20.016515 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:20.017523 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0812 00:58:20.017905 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462}
I20260812 00:58:20.018903 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:19.989498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38635 (local address 127.21.201.66:39265)
0812 00:58:19.993755 (+ 4257us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:19.993784 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:19.994028 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:19.994704 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:19.994712 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:19.995491 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:19.996027 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:19.996046 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:19.997834 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:19.997843 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:20.000115 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:20.000143 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.003252 (+ 3109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.003264 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:20.003537 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:20.006432 (+ 2895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:20.006476 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:20.008600 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:20.013850 (+ 5250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:20.013860 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:20.013866 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:20.014574 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:20.015161 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:20.015166 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:20.015170 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:20.015450 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:20.016524 (+ 1074us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:20.016536 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:20.016879 (+ 343us) client_negotiation.cc:769] Sending connection context
0812 00:58:20.017198 (+ 319us) client_negotiation.cc:240] Negotiation successful
0812 00:58:20.018069 (+ 871us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4062,"thread_start_us":3859,"threads_started":1}
I20260812 00:58:20.021142 25672 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:32843
I20260812 00:58:20.021184 25906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:32843 every 8 connection(s)
I20260812 00:58:20.022894 25672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 00:58:20.027045 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 25672
I20260812 00:58:20.027328 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance
I20260812 00:58:20.027489 25907 heartbeater.cc:347] Connected to a master server at 127.21.201.126:38635
I20260812 00:58:20.027899 25907 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:20.028168 22309 external_mini_cluster.cc:1428] Setting key 1196534a495e473cdf1c8f08d633f7ab
I20260812 00:58:20.029014 25907 heartbeater.cc:511] Master 127.21.201.126:38635 requested a full tablet report, sending...
I20260812 00:58:20.030800 25288 ts_manager.cc:194] Registered new tserver with Master: ad6137c90eb04e17bccecb4fa4977244 (127.21.201.66:32843)
I20260812 00:58:20.031678 25288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.66@KRBTEST.COM'} at 127.21.201.66:46277
WARNING: no policy specified for kudu/127.21.201.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
I20260812 00:58:20.089541 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:0
--local_ip_for_outbound_sockets=127.21.201.67
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38635
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:38389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:58:20.372473 25665 heartbeater.cc:503] Master 127.21.201.126:38635 was elected leader, sending a full tablet report...
W20260812 00:58:20.482036 25912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:20.482472 25912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:20.482666 25912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:20.490679 25912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:20.491066 25912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:20.491323 25912 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 00:58:20.491439 25912 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:58:20.491602 25912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 00:58:20.502753 25912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:38389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.201.126:38635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:20.504597 25912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:20.506136 25912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:20.521595 25921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:20.521777 25922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:20.522748 25927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:20.523468 25912 server_base.cc:1034] running on GCE node
I20260812 00:58:20.524607 25912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:20.533599 25912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:20.534925 25912 hybrid_clock.cc:648] HybridClock initialized: now 1786496300534809 us; error 136 us; skew 500 ppm
I20260812 00:58:20.535519 25912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:20 dist-test-slave-7f01 krb5kdc[25213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496300, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:20.540153 25912 init.cc:377] Logged in from keytab as kudu/127.21.201.67@KRBTEST.COM (short username kudu)
I20260812 00:58:20.542023 25912 webserver.cc:460] Webserver started at http://127.21.201.67:39321/ using document root <none> and password file <none>
I20260812 00:58:20.542829 25912 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:20.542955 25912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:20.543335 25912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:20.546826 25912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-2/data/instance:
uuid: "5848e5a597064322aa7c5b615f34e583"
format_stamp: "Formatted at 2026-08-12 00:58:20 on dist-test-slave-7f01"
server_key: "962821eedb19eb7a93f7eadcac737586"
server_key_iv: "0deb8bbd1dac7e5766f55a466277d6cf"
server_key_version: "encryptionkey@0"
I20260812 00:58:20.547731 25912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance:
uuid: "5848e5a597064322aa7c5b615f34e583"
format_stamp: "Formatted at 2026-08-12 00:58:20 on dist-test-slave-7f01"
server_key: "962821eedb19eb7a93f7eadcac737586"
server_key_iv: "0deb8bbd1dac7e5766f55a466277d6cf"
server_key_version: "encryptionkey@0"
I20260812 00:58:20.554306 25912 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 00:58:20.558743 25937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:20.561250 25912 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 00:58:20.561599 25912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "5848e5a597064322aa7c5b615f34e583"
format_stamp: "Formatted at 2026-08-12 00:58:20 on dist-test-slave-7f01"
server_key: "962821eedb19eb7a93f7eadcac737586"
server_key_iv: "0deb8bbd1dac7e5766f55a466277d6cf"
server_key_version: "encryptionkey@0"
I20260812 00:58:20.561937 25912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:20.599124 25912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:20.604388 25912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:20.604734 25912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:20.608544 25912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:20.608722 25912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:20.608892 25912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:20.608996 25912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 00:58:20 dist-test-slave-7f01 krb5kdc[25213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496300, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:20.642753 25950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:20.611131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38635 (local address 127.21.201.67:36485)
0812 00:58:20.612267 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:20.612331 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:20.615071 (+ 2740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:20.616277 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:20.616291 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:20.617840 (+ 1549us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:20.619393 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:20.619422 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.621571 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.621575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:20.623088 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:20.623104 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.624687 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.624704 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:20.625236 (+ 532us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:20.626624 (+ 1388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:20.626690 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:20.631137 (+ 4447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:20.635968 (+ 4831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:20.635982 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:20.635999 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:20.636867 (+ 868us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:20.637565 (+ 698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:20.637569 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:20.637571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:20.638028 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:20.638870 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:20.638879 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:20.639556 (+ 677us) client_negotiation.cc:769] Sending connection context
0812 00:58:20.640060 (+ 504us) client_negotiation.cc:240] Negotiation successful
0812 00:58:20.640722 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":467,"threads_started":1}
I20260812 00:58:20.643618 25948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:20.609948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:36485 (local address 127.21.201.126:38635)
0812 00:58:20.610788 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:20.610795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:20.612890 (+ 2095us) server_negotiation.cc:407] Connection header received
0812 00:58:20.615522 (+ 2632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:20.615531 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:20.615676 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:20.615874 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:20.619827 (+ 3953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.621306 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.623304 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.624085 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.624256 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:20.631536 (+ 7280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:20.631548 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:20.631556 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:20.631620 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:20.635645 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:20.637170 (+ 1525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:20.637178 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:20.637182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:20.637396 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:20.638191 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:20.638199 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:20.638204 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:20.638694 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:20.638883 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:20.642824 (+ 3941us) server_negotiation.cc:299] Negotiation successful
0812 00:58:20.643159 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":476,"threads_started":1}
I20260812 00:58:20.669199 25948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:20.650960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:50209 (local address 127.21.201.126:38635)
0812 00:58:20.651218 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:20.651227 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:20.651769 (+ 542us) server_negotiation.cc:407] Connection header received
0812 00:58:20.652046 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:20.652050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:20.652149 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:20.652269 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:20.654174 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.655270 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.657933 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.658421 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.658627 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:20.662248 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:20.662254 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:20.662258 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:20.662298 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:20.665823 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:20.666712 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:20.666742 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:20.666747 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:20.666939 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:20.667416 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:20.667422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:20.667424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:20.667845 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:20.668024 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:20.668534 (+ 510us) server_negotiation.cc:299] Negotiation successful
0812 00:58:20.668804 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260812 00:58:20.669397 26118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:20.650914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38635 (local address 127.21.201.67:50209)
0812 00:58:20.651615 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:20.651634 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:20.651881 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:20.652352 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:20.652362 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:20.653228 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:20.653789 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:20.653809 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.655578 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.655588 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:20.657670 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:20.657701 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.658959 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.658964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:20.659171 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:20.660235 (+ 1064us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:20.660270 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:20.661956 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:20.666013 (+ 4057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:20.666023 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:20.666028 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:20.666552 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:20.667067 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:20.667072 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:20.667076 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:20.667296 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:20.667978 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:20.667983 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:20.668220 (+ 237us) client_negotiation.cc:769] Sending connection context
0812 00:58:20.668398 (+ 178us) client_negotiation.cc:240] Negotiation successful
0812 00:58:20.668836 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":408,"threads_started":1}
I20260812 00:58:20.670886 25912 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:40315
I20260812 00:58:20.670920 26168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:40315 every 8 connection(s)
I20260812 00:58:20.672528 25912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 00:58:20.675529 26170 heartbeater.cc:347] Connected to a master server at 127.21.201.126:38635
I20260812 00:58:20.676036 26170 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:20.677333 26170 heartbeater.cc:511] Master 127.21.201.126:38635 requested a full tablet report, sending...
I20260812 00:58:20.679560 25288 ts_manager.cc:194] Registered new tserver with Master: 5848e5a597064322aa7c5b615f34e583 (127.21.201.67:40315)
I20260812 00:58:20.680662 25288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.67@KRBTEST.COM'} at 127.21.201.67:36485
I20260812 00:58:20.681797 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 25912
I20260812 00:58:20.682046 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance
I20260812 00:58:20.682603 22309 external_mini_cluster.cc:1428] Setting key bc020bc4f133c150b9ddc0f686595fac
I20260812 00:58:20.686950 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 00:58:20 dist-test-slave-7f01 krb5kdc[25213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 00:58:20 dist-test-slave-7f01 krb5kdc[25213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:20.755581 25948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:20.734709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58216 (local address 127.21.201.126:38635)
0812 00:58:20.734946 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:20.734951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:20.735374 (+ 423us) server_negotiation.cc:407] Connection header received
0812 00:58:20.735483 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:20.735488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:20.735563 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:20.735651 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:20.737466 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.738269 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.740222 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.740797 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.740947 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:20.746736 (+ 5789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:20.746744 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:20.746748 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:20.746827 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:20.751024 (+ 4197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:20.752416 (+ 1392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:20.752423 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:20.752426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:20.752656 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:20.753253 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:20.753258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:20.753260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:20.753610 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:20.753847 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:20.754731 (+ 884us) server_negotiation.cc:299] Negotiation successful
0812 00:58:20.755191 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260812 00:58:20.761588 25288 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:58:20.770996 25288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:58:20.808465 26198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:20.788806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34662 (local address 127.21.201.66:32843)
0812 00:58:20.796827 (+ 8021us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:20.796841 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:20.798849 (+ 2008us) server_negotiation.cc:407] Connection header received
0812 00:58:20.799011 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:20.799021 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:20.799269 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:20.799703 (+ 434us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:58:20.800974 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.801962 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.804951 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.806087 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.806235 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:20.806593 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:20.807891 (+ 1298us) server_negotiation.cc:299] Negotiation successful
0812 00:58:20.808103 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7770,"thread_start_us":7602,"threads_started":1}
I20260812 00:58:20.809243 26190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:20.787533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:32843 (local address 127.0.0.1:34662)
0812 00:58:20.798479 (+ 10946us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:20.798509 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:20.798683 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:20.799508 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:20.799516 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:20.799549 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:58:20.800748 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:20.800762 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.802104 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.802113 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:20.804758 (+ 2645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:20.804774 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.806911 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.806919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:20.807172 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:20.807231 (+ 59us) client_negotiation.cc:769] Sending connection context
0812 00:58:20.807370 (+ 139us) client_negotiation.cc:240] Negotiation successful
0812 00:58:20.807614 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10773,"mutex_wait_us":89,"spinlock_wait_cycles":1408,"thread_start_us":10630,"threads_started":1}
I20260812 00:58:20.809451 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:20.787668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58774 (local address 127.21.201.65:44581)
0812 00:58:20.798047 (+ 10379us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:20.798060 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:20.798102 (+ 42us) server_negotiation.cc:407] Connection header received
0812 00:58:20.798244 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:20.798255 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:20.798593 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:20.798832 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:58:20.800281 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.802723 (+ 2442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.805821 (+ 3098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.807542 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.807694 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:20.807936 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:20.808539 (+ 603us) server_negotiation.cc:299] Negotiation successful
0812 00:58:20.809016 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10191,"thread_start_us":10043,"threads_started":1}
I20260812 00:58:20.809595 26197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:20.787483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:44581 (local address 127.0.0.1:58774)
0812 00:58:20.795594 (+ 8111us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:20.795665 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:20.795998 (+ 333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:20.799107 (+ 3109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:20.799116 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:20.799155 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:58:20.799758 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:20.799775 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.802913 (+ 3138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.802919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:20.805630 (+ 2711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:20.805640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.808069 (+ 2429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.808074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:20.808305 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:20.808348 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 00:58:20.808441 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 00:58:20.808603 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7677,"mutex_wait_us":281,"thread_start_us":11142,"threads_started":1}
I20260812 00:58:20.809733 26192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:20.788071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:40315 (local address 127.0.0.1:36938)
0812 00:58:20.798956 (+ 10885us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:20.798975 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:20.799104 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:20.799868 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:20.799872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:20.799899 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:58:20.800846 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:20.800859 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.802895 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.802901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:20.805618 (+ 2717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:20.805635 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.808017 (+ 2382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.808023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:20.808223 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:20.808259 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 00:58:20.808363 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 00:58:20.808512 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10752,"mutex_wait_us":133,"thread_start_us":6878,"threads_started":1}
I20260812 00:58:20.811492 26199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:20.789624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36938 (local address 127.21.201.67:40315)
0812 00:58:20.797558 (+ 7934us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:20.797569 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:20.799251 (+ 1682us) server_negotiation.cc:407] Connection header received
0812 00:58:20.799390 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:20.799398 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:20.799598 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:20.800255 (+ 657us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:58:20.801105 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.802690 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.805850 (+ 3160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.807480 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.807604 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:20.807866 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:20.808682 (+ 816us) server_negotiation.cc:299] Negotiation successful
0812 00:58:20.808853 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7700,"thread_start_us":7468,"threads_started":1}
I20260812 00:58:20.813517 25774 tablet_service.cc:1467] Processing CreateTablet for tablet 35a33836aedf45f9a1b6b84f9eb9125f (DEFAULT_TABLE table=test-table [id=392776d93a4c470498eeed8684fc1c33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:20.813867 25540 tablet_service.cc:1467] Processing CreateTablet for tablet 35a33836aedf45f9a1b6b84f9eb9125f (DEFAULT_TABLE table=test-table [id=392776d93a4c470498eeed8684fc1c33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:20.815035 25774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35a33836aedf45f9a1b6b84f9eb9125f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:20.815300 25540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35a33836aedf45f9a1b6b84f9eb9125f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:20.815243 26044 tablet_service.cc:1467] Processing CreateTablet for tablet 35a33836aedf45f9a1b6b84f9eb9125f (DEFAULT_TABLE table=test-table [id=392776d93a4c470498eeed8684fc1c33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:20.816798 26044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35a33836aedf45f9a1b6b84f9eb9125f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:20.828830 26201 tablet_bootstrap.cc:492] T 35a33836aedf45f9a1b6b84f9eb9125f P ad6137c90eb04e17bccecb4fa4977244: Bootstrap starting.
I20260812 00:58:20.834009 26201 tablet_bootstrap.cc:654] T 35a33836aedf45f9a1b6b84f9eb9125f P ad6137c90eb04e17bccecb4fa4977244: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:20.835394 26201 log.cc:824] T 35a33836aedf45f9a1b6b84f9eb9125f P ad6137c90eb04e17bccecb4fa4977244: Log is configured to *not* fsync() on all Append() calls
I20260812 00:58:20.836774 26205 tablet_bootstrap.cc:492] T 35a33836aedf45f9a1b6b84f9eb9125f P 5848e5a597064322aa7c5b615f34e583: Bootstrap starting.
I20260812 00:58:20.837138 26204 tablet_bootstrap.cc:492] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28: Bootstrap starting.
I20260812 00:58:20.841308 26201 tablet_bootstrap.cc:492] T 35a33836aedf45f9a1b6b84f9eb9125f P ad6137c90eb04e17bccecb4fa4977244: No bootstrap required, opened a new log
I20260812 00:58:20.841274 26204 tablet_bootstrap.cc:654] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:20.841288 26205 tablet_bootstrap.cc:654] T 35a33836aedf45f9a1b6b84f9eb9125f P 5848e5a597064322aa7c5b615f34e583: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:20.841648 26201 ts_tablet_manager.cc:1397] T 35a33836aedf45f9a1b6b84f9eb9125f P ad6137c90eb04e17bccecb4fa4977244: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260812 00:58:20.842634 26205 log.cc:824] T 35a33836aedf45f9a1b6b84f9eb9125f P 5848e5a597064322aa7c5b615f34e583: Log is configured to *not* fsync() on all Append() calls
I20260812 00:58:20.842733 26204 log.cc:824] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28: Log is configured to *not* fsync() on all Append() calls
I20260812 00:58:20.846457 26204 tablet_bootstrap.cc:492] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28: No bootstrap required, opened a new log
I20260812 00:58:20.846561 26205 tablet_bootstrap.cc:492] T 35a33836aedf45f9a1b6b84f9eb9125f P 5848e5a597064322aa7c5b615f34e583: No bootstrap required, opened a new log
I20260812 00:58:20.846853 26204 ts_tablet_manager.cc:1397] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260812 00:58:20.847288 26205 ts_tablet_manager.cc:1397] T 35a33836aedf45f9a1b6b84f9eb9125f P 5848e5a597064322aa7c5b615f34e583: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260812 00:58:20.855082 26201 raft_consensus.cc:348] T 35a33836aedf45f9a1b6b84f9eb9125f P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:20.855681 26201 raft_consensus.cc:374] T 35a33836aedf45f9a1b6b84f9eb9125f P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:20.855820 26201 raft_consensus.cc:729] T 35a33836aedf45f9a1b6b84f9eb9125f P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6137c90eb04e17bccecb4fa4977244, State: Initialized, Role: FOLLOWER
I20260812 00:58:20.855515 26204 raft_consensus.cc:348] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:20.856580 26204 raft_consensus.cc:374] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:20.856739 26204 raft_consensus.cc:729] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dbc61bb753e46a9be28f8e789ad8b28, State: Initialized, Role: FOLLOWER
I20260812 00:58:20.856802 26201 consensus_queue.cc:260] T 35a33836aedf45f9a1b6b84f9eb9125f P ad6137c90eb04e17bccecb4fa4977244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:20.857554 26204 consensus_queue.cc:260] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:20.858719 25907 heartbeater.cc:503] Master 127.21.201.126:38635 was elected leader, sending a full tablet report...
I20260812 00:58:20.859776 26201 ts_tablet_manager.cc:1428] T 35a33836aedf45f9a1b6b84f9eb9125f P ad6137c90eb04e17bccecb4fa4977244: Time spent starting tablet: real 0.018s user 0.018s sys 0.000s
I20260812 00:58:20.860241 26205 raft_consensus.cc:348] T 35a33836aedf45f9a1b6b84f9eb9125f P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:20.860797 26205 raft_consensus.cc:374] T 35a33836aedf45f9a1b6b84f9eb9125f P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:20.861680 26205 raft_consensus.cc:729] T 35a33836aedf45f9a1b6b84f9eb9125f P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5848e5a597064322aa7c5b615f34e583, State: Initialized, Role: FOLLOWER
I20260812 00:58:20.861737 26204 ts_tablet_manager.cc:1428] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
W20260812 00:58:20.862857 25667 tablet.cc:2369] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:58:20.862511 26205 consensus_queue.cc:260] T 35a33836aedf45f9a1b6b84f9eb9125f P 5848e5a597064322aa7c5b615f34e583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:20.865195 26170 heartbeater.cc:503] Master 127.21.201.126:38635 was elected leader, sending a full tablet report...
I20260812 00:58:20.866080 26205 ts_tablet_manager.cc:1428] T 35a33836aedf45f9a1b6b84f9eb9125f P 5848e5a597064322aa7c5b615f34e583: Time spent starting tablet: real 0.019s user 0.019s sys 0.000s
I20260812 00:58:20.897749 26213 raft_consensus.cc:482] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:58:20.898196 26213 raft_consensus.cc:504] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:20.900189 26213 leader_election.cc:288] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad6137c90eb04e17bccecb4fa4977244 (127.21.201.66:32843), 5848e5a597064322aa7c5b615f34e583 (127.21.201.67:40315)
I20260812 00:58:20.911886 26199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:20.901028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:50645 (local address 127.21.201.67:40315)
0812 00:58:20.901289 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:20.901298 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:20.901938 (+ 640us) server_negotiation.cc:407] Connection header received
0812 00:58:20.902185 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:20.902190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:20.902265 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:20.902363 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:58:20.903715 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.904753 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.908779 (+ 4026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.910096 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.910364 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:20.910571 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:20.911272 (+ 701us) server_negotiation.cc:299] Negotiation successful
0812 00:58:20.911486 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260812 00:58:20.911887 26198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:20.900851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:34167 (local address 127.21.201.66:32843)
0812 00:58:20.901094 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:20.901104 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:20.901938 (+ 834us) server_negotiation.cc:407] Connection header received
0812 00:58:20.902155 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:20.902161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:20.902244 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:20.902363 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:58:20.903784 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.904811 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.908637 (+ 3826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.910117 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.910351 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:20.910560 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:20.911272 (+ 712us) server_negotiation.cc:299] Negotiation successful
0812 00:58:20.911494 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 00:58:20.912235 26219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:20.900703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:40315 (local address 127.21.201.65:50645)
0812 00:58:20.901748 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:20.901768 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:20.901958 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:20.902554 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:20.902563 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:20.902652 (+ 89us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:58:20.903417 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:20.903436 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.904972 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.904978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:20.908252 (+ 3274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:20.908275 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.910618 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.910625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:20.910880 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:20.910968 (+ 88us) client_negotiation.cc:769] Sending connection context
0812 00:58:20.911102 (+ 134us) client_negotiation.cc:240] Negotiation successful
0812 00:58:20.911339 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"mutex_wait_us":354,"spinlock_wait_cycles":27264,"thread_start_us":549,"threads_started":1}
I20260812 00:58:20.912595 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:20.900609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:32843 (local address 127.21.201.65:34167)
0812 00:58:20.901729 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:20.901756 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:20.901958 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:20.902554 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:20.902563 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:20.902653 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:58:20.903426 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:20.903436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.905037 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.905044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:20.908251 (+ 3207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:20.908275 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.910601 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.910609 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:20.910875 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:20.910982 (+ 107us) client_negotiation.cc:769] Sending connection context
0812 00:58:20.911102 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 00:58:20.911339 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"mutex_wait_us":388,"spinlock_wait_cycles":143104,"thread_start_us":872,"threads_started":1}
I20260812 00:58:20.913993 25815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35a33836aedf45f9a1b6b84f9eb9125f" candidate_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6137c90eb04e17bccecb4fa4977244" is_pre_election: true
I20260812 00:58:20.913982 26085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35a33836aedf45f9a1b6b84f9eb9125f" candidate_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5848e5a597064322aa7c5b615f34e583" is_pre_election: true
I20260812 00:58:20.914633 25815 raft_consensus.cc:2454] T 35a33836aedf45f9a1b6b84f9eb9125f P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dbc61bb753e46a9be28f8e789ad8b28 in term 0.
I20260812 00:58:20.914649 26085 raft_consensus.cc:2454] T 35a33836aedf45f9a1b6b84f9eb9125f P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dbc61bb753e46a9be28f8e789ad8b28 in term 0.
I20260812 00:58:20.915499 25450 leader_election.cc:302] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0dbc61bb753e46a9be28f8e789ad8b28, 5848e5a597064322aa7c5b615f34e583, ad6137c90eb04e17bccecb4fa4977244; no voters:
I20260812 00:58:20.916064 26213 raft_consensus.cc:2781] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:58:20.916514 26213 raft_consensus.cc:482] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:58:20.916745 26213 raft_consensus.cc:3037] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:20.919890 26213 raft_consensus.cc:504] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:20.920969 26213 leader_election.cc:288] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28 [CANDIDATE]: Term 1 election: Requested vote from peers ad6137c90eb04e17bccecb4fa4977244 (127.21.201.66:32843), 5848e5a597064322aa7c5b615f34e583 (127.21.201.67:40315)
I20260812 00:58:20.922065 26085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35a33836aedf45f9a1b6b84f9eb9125f" candidate_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5848e5a597064322aa7c5b615f34e583"
I20260812 00:58:20.922096 25815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35a33836aedf45f9a1b6b84f9eb9125f" candidate_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6137c90eb04e17bccecb4fa4977244"
I20260812 00:58:20.922384 26085 raft_consensus.cc:3037] T 35a33836aedf45f9a1b6b84f9eb9125f P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:20.922405 25815 raft_consensus.cc:3037] T 35a33836aedf45f9a1b6b84f9eb9125f P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Advancing to term 1
W20260812 00:58:20.924721 26172 tablet.cc:2369] T 35a33836aedf45f9a1b6b84f9eb9125f P 5848e5a597064322aa7c5b615f34e583: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:58:20.925349 25815 raft_consensus.cc:2454] T 35a33836aedf45f9a1b6b84f9eb9125f P ad6137c90eb04e17bccecb4fa4977244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dbc61bb753e46a9be28f8e789ad8b28 in term 1.
I20260812 00:58:20.926025 25455 leader_election.cc:302] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dbc61bb753e46a9be28f8e789ad8b28, ad6137c90eb04e17bccecb4fa4977244; no voters:
I20260812 00:58:20.926244 26085 raft_consensus.cc:2454] T 35a33836aedf45f9a1b6b84f9eb9125f P 5848e5a597064322aa7c5b615f34e583 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dbc61bb753e46a9be28f8e789ad8b28 in term 1.
I20260812 00:58:20.926542 26213 raft_consensus.cc:2781] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:58:20.927222 26213 raft_consensus.cc:686] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28 [term 1 LEADER]: Becoming Leader. State: Replica: 0dbc61bb753e46a9be28f8e789ad8b28, State: Running, Role: LEADER
I20260812 00:58:20.927915 26213 consensus_queue.cc:237] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:20.932911 25285 catalog_manager.cc:5495] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dbc61bb753e46a9be28f8e789ad8b28 (127.21.201.65). New cstate: current_term: 1 leader_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } health_report { overall_health: UNKNOWN } } }
Aug 12 00:58:20 dist-test-slave-7f01 krb5kdc[25213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496298, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:21.001531 25948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:20.982786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58222 (local address 127.21.201.126:38635)
0812 00:58:20.983063 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:20.983071 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:20.983804 (+ 733us) server_negotiation.cc:407] Connection header received
0812 00:58:20.983985 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:20.983989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:20.984053 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:20.984154 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:20.985700 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.986584 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.988238 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.988748 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.988884 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:20.994163 (+ 5279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:20.994171 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:20.994175 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:20.994225 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:20.997821 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:20.998998 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:20.999003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:20.999005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:20.999150 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:20.999914 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:20.999918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:20.999920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:21.000256 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:21.000412 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:21.001044 (+ 632us) server_negotiation.cc:299] Negotiation successful
0812 00:58:21.001235 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260812 00:58:21.001538 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:20.982908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38635 (local address 127.0.0.1:58222)
0812 00:58:20.983611 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:20.983629 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:20.983848 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:20.984220 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:20.984224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:20.984991 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:20.985300 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:20.985311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.986823 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.986836 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:20.988036 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:20.988052 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:20.989028 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:20.989033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:20.989150 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:20.990129 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:20.990164 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:20.993746 (+ 3582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:20.998072 (+ 4326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:20.998094 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:20.998101 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:20.998833 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:20.999376 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:20.999382 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:20.999386 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:20.999665 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:21.000482 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:21.000490 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:21.000671 (+ 181us) client_negotiation.cc:769] Sending connection context
0812 00:58:21.000825 (+ 154us) client_negotiation.cc:240] Negotiation successful
0812 00:58:21.001095 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":415,"threads_started":1}
I20260812 00:58:21.008339 25285 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:58222:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 00:58:21.009928 25285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:58:21.019666 25774 tablet_service.cc:1467] Processing CreateTablet for tablet d04e40d6fdd843e98623b435b00f9fa5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50fe87fe2a7b41c68fca55241adacd63]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 00:58:21.019563 25540 tablet_service.cc:1467] Processing CreateTablet for tablet d04e40d6fdd843e98623b435b00f9fa5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50fe87fe2a7b41c68fca55241adacd63]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 00:58:21.020991 25774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d04e40d6fdd843e98623b435b00f9fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:21.021006 25540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d04e40d6fdd843e98623b435b00f9fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:21.025416 26044 tablet_service.cc:1467] Processing CreateTablet for tablet d04e40d6fdd843e98623b435b00f9fa5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50fe87fe2a7b41c68fca55241adacd63]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
W20260812 00:58:21.026566 25908 tablet.cc:2369] T 35a33836aedf45f9a1b6b84f9eb9125f P ad6137c90eb04e17bccecb4fa4977244: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:58:21.026842 26044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d04e40d6fdd843e98623b435b00f9fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:21.030140 26204 tablet_bootstrap.cc:492] T d04e40d6fdd843e98623b435b00f9fa5 P 0dbc61bb753e46a9be28f8e789ad8b28: Bootstrap starting.
I20260812 00:58:21.032312 26204 tablet_bootstrap.cc:654] T d04e40d6fdd843e98623b435b00f9fa5 P 0dbc61bb753e46a9be28f8e789ad8b28: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:21.035897 26201 tablet_bootstrap.cc:492] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244: Bootstrap starting.
I20260812 00:58:21.036355 26204 tablet_bootstrap.cc:492] T d04e40d6fdd843e98623b435b00f9fa5 P 0dbc61bb753e46a9be28f8e789ad8b28: No bootstrap required, opened a new log
I20260812 00:58:21.036540 26204 ts_tablet_manager.cc:1397] T d04e40d6fdd843e98623b435b00f9fa5 P 0dbc61bb753e46a9be28f8e789ad8b28: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 00:58:21.037520 26204 raft_consensus.cc:348] T d04e40d6fdd843e98623b435b00f9fa5 P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:21.037858 26204 raft_consensus.cc:374] T d04e40d6fdd843e98623b435b00f9fa5 P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:21.037974 26204 raft_consensus.cc:729] T d04e40d6fdd843e98623b435b00f9fa5 P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dbc61bb753e46a9be28f8e789ad8b28, State: Initialized, Role: FOLLOWER
I20260812 00:58:21.038250 26204 consensus_queue.cc:260] T d04e40d6fdd843e98623b435b00f9fa5 P 0dbc61bb753e46a9be28f8e789ad8b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:21.038390 26201 tablet_bootstrap.cc:654] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:21.038941 26205 tablet_bootstrap.cc:492] T d04e40d6fdd843e98623b435b00f9fa5 P 5848e5a597064322aa7c5b615f34e583: Bootstrap starting.
I20260812 00:58:21.038858 26213 tablet_replica.cc:439] T d04e40d6fdd843e98623b435b00f9fa5 P 0dbc61bb753e46a9be28f8e789ad8b28: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } } }
I20260812 00:58:21.039427 26213 tablet_replica.cc:442] T d04e40d6fdd843e98623b435b00f9fa5 P 0dbc61bb753e46a9be28f8e789ad8b28: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:58:21.041128 26204 ts_tablet_manager.cc:1428] T d04e40d6fdd843e98623b435b00f9fa5 P 0dbc61bb753e46a9be28f8e789ad8b28: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260812 00:58:21.041237 26205 tablet_bootstrap.cc:654] T d04e40d6fdd843e98623b435b00f9fa5 P 5848e5a597064322aa7c5b615f34e583: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:21.041692 26201 tablet_bootstrap.cc:492] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244: No bootstrap required, opened a new log
I20260812 00:58:21.041893 26201 ts_tablet_manager.cc:1397] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260812 00:58:21.042861 26201 raft_consensus.cc:348] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:21.043273 26201 raft_consensus.cc:374] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:21.043411 26201 raft_consensus.cc:729] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6137c90eb04e17bccecb4fa4977244, State: Initialized, Role: FOLLOWER
I20260812 00:58:21.043789 26201 consensus_queue.cc:260] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:21.044734 26201 ts_tablet_manager.cc:1428] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 00:58:21.044457 26212 tablet_replica.cc:439] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } } }
I20260812 00:58:21.044945 26212 tablet_replica.cc:442] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:58:21.049510 26205 tablet_bootstrap.cc:492] T d04e40d6fdd843e98623b435b00f9fa5 P 5848e5a597064322aa7c5b615f34e583: No bootstrap required, opened a new log
I20260812 00:58:21.049716 26205 ts_tablet_manager.cc:1397] T d04e40d6fdd843e98623b435b00f9fa5 P 5848e5a597064322aa7c5b615f34e583: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260812 00:58:21.050617 26205 raft_consensus.cc:348] T d04e40d6fdd843e98623b435b00f9fa5 P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:21.050843 26205 raft_consensus.cc:374] T d04e40d6fdd843e98623b435b00f9fa5 P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:21.050958 26205 raft_consensus.cc:729] T d04e40d6fdd843e98623b435b00f9fa5 P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5848e5a597064322aa7c5b615f34e583, State: Initialized, Role: FOLLOWER
I20260812 00:58:21.051229 26205 consensus_queue.cc:260] T d04e40d6fdd843e98623b435b00f9fa5 P 5848e5a597064322aa7c5b615f34e583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:21.052109 26205 ts_tablet_manager.cc:1428] T d04e40d6fdd843e98623b435b00f9fa5 P 5848e5a597064322aa7c5b615f34e583: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260812 00:58:21.052079 26216 tablet_replica.cc:439] T d04e40d6fdd843e98623b435b00f9fa5 P 5848e5a597064322aa7c5b615f34e583: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } } }
I20260812 00:58:21.052685 26216 tablet_replica.cc:442] T d04e40d6fdd843e98623b435b00f9fa5 P 5848e5a597064322aa7c5b615f34e583: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:58:21.053330 26212 raft_consensus.cc:482] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:58:21.053603 26212 raft_consensus.cc:504] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:21.056732 26212 leader_election.cc:288] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dbc61bb753e46a9be28f8e789ad8b28 (127.21.201.65:44581), 5848e5a597064322aa7c5b615f34e583 (127.21.201.67:40315)
I20260812 00:58:21.066177 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:21.057629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:59579 (local address 127.21.201.65:44581)
0812 00:58:21.057999 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:21.058004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:21.060553 (+ 2549us) server_negotiation.cc:407] Connection header received
0812 00:58:21.060780 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:21.060786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:21.060867 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:21.060977 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:58:21.061556 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.062336 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.064269 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.065170 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.065376 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:21.065461 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:21.065752 (+ 291us) server_negotiation.cc:299] Negotiation successful
0812 00:58:21.065893 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220}
I20260812 00:58:21.066628 26242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:21.056877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:44581 (local address 127.21.201.66:59579)
0812 00:58:21.060422 (+ 3545us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:21.060441 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:21.060632 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:21.060996 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:21.061001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:21.061025 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:58:21.061396 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:21.061405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.062492 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.062497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:21.064103 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:21.064119 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.065405 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.065412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:21.065582 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:21.065607 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 00:58:21.065722 (+ 115us) client_negotiation.cc:240] Negotiation successful
0812 00:58:21.065880 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3395,"thread_start_us":3204,"threads_started":1}
I20260812 00:58:21.067920 25566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d04e40d6fdd843e98623b435b00f9fa5" candidate_uuid: "ad6137c90eb04e17bccecb4fa4977244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" is_pre_election: true
I20260812 00:58:21.068370 25566 raft_consensus.cc:2454] T d04e40d6fdd843e98623b435b00f9fa5 P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad6137c90eb04e17bccecb4fa4977244 in term 0.
I20260812 00:58:21.069579 25696 leader_election.cc:302] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dbc61bb753e46a9be28f8e789ad8b28, ad6137c90eb04e17bccecb4fa4977244; no voters:
I20260812 00:58:21.070288 26212 raft_consensus.cc:2781] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:58:21.070498 26212 raft_consensus.cc:482] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:58:21.070616 26212 raft_consensus.cc:3037] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:21.072981 26243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:21.057272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:40315 (local address 127.21.201.66:52461)
0812 00:58:21.065450 (+ 8178us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:21.065469 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:21.065649 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:21.066038 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:21.066042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:21.066067 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:58:21.066796 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:21.066810 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.068394 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.068402 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:21.070214 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:21.070234 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.071811 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.071820 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:21.072089 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:21.072142 (+ 53us) client_negotiation.cc:769] Sending connection context
0812 00:58:21.072320 (+ 178us) client_negotiation.cc:240] Negotiation successful
0812 00:58:21.072537 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7958,"thread_start_us":7263,"threads_started":1}
I20260812 00:58:21.072971 26199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:21.057689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:52461 (local address 127.21.201.67:40315)
0812 00:58:21.057891 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:21.057899 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:21.065586 (+ 7687us) server_negotiation.cc:407] Connection header received
0812 00:58:21.065793 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:21.065798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:21.065907 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:21.066020 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:58:21.067075 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.068256 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.070491 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.071413 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.071568 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:21.071627 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:21.072416 (+ 789us) server_negotiation.cc:299] Negotiation successful
0812 00:58:21.072693 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 00:58:21.074430 26085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d04e40d6fdd843e98623b435b00f9fa5" candidate_uuid: "ad6137c90eb04e17bccecb4fa4977244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5848e5a597064322aa7c5b615f34e583" is_pre_election: true
I20260812 00:58:21.074419 26212 raft_consensus.cc:504] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:21.074815 26085 raft_consensus.cc:2454] T d04e40d6fdd843e98623b435b00f9fa5 P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad6137c90eb04e17bccecb4fa4977244 in term 0.
I20260812 00:58:21.075306 26212 leader_election.cc:288] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244 [CANDIDATE]: Term 1 election: Requested vote from peers 0dbc61bb753e46a9be28f8e789ad8b28 (127.21.201.65:44581), 5848e5a597064322aa7c5b615f34e583 (127.21.201.67:40315)
I20260812 00:58:21.076089 26085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d04e40d6fdd843e98623b435b00f9fa5" candidate_uuid: "ad6137c90eb04e17bccecb4fa4977244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5848e5a597064322aa7c5b615f34e583"
I20260812 00:58:21.076355 26085 raft_consensus.cc:3037] T d04e40d6fdd843e98623b435b00f9fa5 P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:21.076308 25566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d04e40d6fdd843e98623b435b00f9fa5" candidate_uuid: "ad6137c90eb04e17bccecb4fa4977244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dbc61bb753e46a9be28f8e789ad8b28"
I20260812 00:58:21.076650 25566 raft_consensus.cc:3037] T d04e40d6fdd843e98623b435b00f9fa5 P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:21.078912 26085 raft_consensus.cc:2454] T d04e40d6fdd843e98623b435b00f9fa5 P 5848e5a597064322aa7c5b615f34e583 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad6137c90eb04e17bccecb4fa4977244 in term 1.
I20260812 00:58:21.079460 25566 raft_consensus.cc:2454] T d04e40d6fdd843e98623b435b00f9fa5 P 0dbc61bb753e46a9be28f8e789ad8b28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad6137c90eb04e17bccecb4fa4977244 in term 1.
I20260812 00:58:21.079576 25699 leader_election.cc:302] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5848e5a597064322aa7c5b615f34e583, ad6137c90eb04e17bccecb4fa4977244; no voters:
I20260812 00:58:21.080068 26212 raft_consensus.cc:2781] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:58:21.080636 26212 raft_consensus.cc:686] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244 [term 1 LEADER]: Becoming Leader. State: Replica: ad6137c90eb04e17bccecb4fa4977244, State: Running, Role: LEADER
I20260812 00:58:21.081362 26212 consensus_queue.cc:237] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:21.085420 26246 tablet_replica.cc:439] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244: TxnStatusTablet state changed. Reason: New leader ad6137c90eb04e17bccecb4fa4977244. Latest consensus state: current_term: 1 leader_uuid: "ad6137c90eb04e17bccecb4fa4977244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } } }
I20260812 00:58:21.085937 26246 tablet_replica.cc:442] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:58:21.086803 26250 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 00:58:21.087116 26250 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 00:58:21.086988 25285 catalog_manager.cc:5495] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad6137c90eb04e17bccecb4fa4977244 (127.21.201.66). New cstate: current_term: 1 leader_uuid: "ad6137c90eb04e17bccecb4fa4977244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } health_report { overall_health: UNKNOWN } } }
I20260812 00:58:21.115785 26198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:21.107645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34678 (local address 127.21.201.66:32843)
0812 00:58:21.107852 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:21.107857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:21.107933 (+ 76us) server_negotiation.cc:407] Connection header received
0812 00:58:21.108211 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:21.108217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:21.108330 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:21.108533 (+ 203us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:58:21.108904 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.110190 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.112045 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.112826 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.112994 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:21.113910 (+ 916us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:58:21.114809 (+ 899us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:58:21.114986 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:21.115217 (+ 231us) server_negotiation.cc:299] Negotiation successful
0812 00:58:21.115349 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 00:58:21.115689 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:21.107516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:32843 (local address 127.0.0.1:34678)
0812 00:58:21.107828 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:21.107846 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:21.107993 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:21.108582 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:21.108585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:21.108596 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:58:21.108763 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:21.108771 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.110483 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.110490 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:21.111709 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:21.111730 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.113188 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.113206 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:21.113434 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:21.113556 (+ 122us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:58:21.115008 (+ 1452us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:58:21.115041 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 00:58:21.115150 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 00:58:21.115364 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 00:58:21.127147 25566 raft_consensus.cc:1261] T d04e40d6fdd843e98623b435b00f9fa5 P 0dbc61bb753e46a9be28f8e789ad8b28 [term 1 FOLLOWER]: Refusing update from remote peer ad6137c90eb04e17bccecb4fa4977244: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:58:21.127329 26085 raft_consensus.cc:1261] T d04e40d6fdd843e98623b435b00f9fa5 P 5848e5a597064322aa7c5b615f34e583 [term 1 FOLLOWER]: Refusing update from remote peer ad6137c90eb04e17bccecb4fa4977244: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:58:21.128563 26212 consensus_queue.cc:1035] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:58:21.128981 26246 consensus_queue.cc:1035] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:58:21.141040 26216 tablet_replica.cc:439] T d04e40d6fdd843e98623b435b00f9fa5 P 5848e5a597064322aa7c5b615f34e583: TxnStatusTablet state changed. Reason: New leader ad6137c90eb04e17bccecb4fa4977244. Latest consensus state: current_term: 1 leader_uuid: "ad6137c90eb04e17bccecb4fa4977244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } } }
I20260812 00:58:21.141494 26216 tablet_replica.cc:442] T d04e40d6fdd843e98623b435b00f9fa5 P 5848e5a597064322aa7c5b615f34e583: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:58:21.145979 26213 tablet_replica.cc:439] T d04e40d6fdd843e98623b435b00f9fa5 P 0dbc61bb753e46a9be28f8e789ad8b28: TxnStatusTablet state changed. Reason: New leader ad6137c90eb04e17bccecb4fa4977244. Latest consensus state: current_term: 1 leader_uuid: "ad6137c90eb04e17bccecb4fa4977244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } } }
I20260812 00:58:21.146517 26213 tablet_replica.cc:442] T d04e40d6fdd843e98623b435b00f9fa5 P 0dbc61bb753e46a9be28f8e789ad8b28: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:58:21.149791 26259 mvcc.cc:205] Tried to move back new op lower bound from 7317488849403035648 to 7317488849233412096. Current Snapshot: MvccSnapshot[applied={T|T < 7317488849403035648}]
I20260812 00:58:21.151495 26216 tablet_replica.cc:439] T d04e40d6fdd843e98623b435b00f9fa5 P 5848e5a597064322aa7c5b615f34e583: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ad6137c90eb04e17bccecb4fa4977244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } } }
I20260812 00:58:21.151952 26216 tablet_replica.cc:442] T d04e40d6fdd843e98623b435b00f9fa5 P 5848e5a597064322aa7c5b615f34e583: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:58:21.161187 26212 tablet_replica.cc:439] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ad6137c90eb04e17bccecb4fa4977244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } } }
I20260812 00:58:21.161621 26212 tablet_replica.cc:442] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:58:21.162876 26246 tablet_replica.cc:439] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ad6137c90eb04e17bccecb4fa4977244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } } }
I20260812 00:58:21.163178 26246 tablet_replica.cc:442] T d04e40d6fdd843e98623b435b00f9fa5 P ad6137c90eb04e17bccecb4fa4977244: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:58:21.166512 26261 mvcc.cc:205] Tried to move back new op lower bound from 7317488849403035648 to 7317488849233412096. Current Snapshot: MvccSnapshot[applied={T|T < 7317488849403035648}]
I20260812 00:58:21.167948 26213 tablet_replica.cc:439] T d04e40d6fdd843e98623b435b00f9fa5 P 0dbc61bb753e46a9be28f8e789ad8b28: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ad6137c90eb04e17bccecb4fa4977244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } } }
I20260812 00:58:21.168503 26213 tablet_replica.cc:442] T d04e40d6fdd843e98623b435b00f9fa5 P 0dbc61bb753e46a9be28f8e789ad8b28: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:58:21.181474 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:21.173717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58788 (local address 127.21.201.65:44581)
0812 00:58:21.173920 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:21.173925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:21.174088 (+ 163us) server_negotiation.cc:407] Connection header received
0812 00:58:21.174349 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:21.174355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:21.174503 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:21.174739 (+ 236us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:58:21.175411 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.176691 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.178718 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.179203 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.179379 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:21.179957 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:58:21.180728 (+ 771us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:58:21.180907 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:21.180998 (+ 91us) server_negotiation.cc:299] Negotiation successful
0812 00:58:21.181106 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 00:58:21.205163 26198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:21.197725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:48231 (local address 127.21.201.66:32843)
0812 00:58:21.197967 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:21.197974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:21.198507 (+ 533us) server_negotiation.cc:407] Connection header received
0812 00:58:21.198905 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:21.198911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:21.199025 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:21.199190 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:58:21.199952 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.200794 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.203045 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.203603 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.203784 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:21.204438 (+ 654us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:58:21.204562 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:58:21.204636 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:21.204812 (+ 176us) server_negotiation.cc:299] Negotiation successful
0812 00:58:21.204889 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 00:58:21.205420 26275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:21.197396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:32843 (local address 127.21.201.65:48231)
0812 00:58:21.198355 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:21.198377 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:21.198574 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:21.199239 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:21.199245 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:21.199260 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:58:21.199680 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:21.199693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.200942 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.200950 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:21.202802 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:21.202821 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.203862 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.203869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:21.204059 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:21.204185 (+ 126us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:58:21.204679 (+ 494us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:58:21.204692 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 00:58:21.204784 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 00:58:21.204956 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":352,"threads_started":1}
I20260812 00:58:21.221225 25815 raft_consensus.cc:1261] T 35a33836aedf45f9a1b6b84f9eb9125f P ad6137c90eb04e17bccecb4fa4977244 [term 1 FOLLOWER]: Refusing update from remote peer 0dbc61bb753e46a9be28f8e789ad8b28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:58:21.222210 26085 raft_consensus.cc:1261] T 35a33836aedf45f9a1b6b84f9eb9125f P 5848e5a597064322aa7c5b615f34e583 [term 1 FOLLOWER]: Refusing update from remote peer 0dbc61bb753e46a9be28f8e789ad8b28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:58:21.222482 26223 consensus_queue.cc:1035] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:58:21.223024 26213 consensus_queue.cc:1035] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:58:21.270300 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:21.262179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:51573 (local address 127.21.201.65:44581)
0812 00:58:21.262416 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:21.262423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:21.262920 (+ 497us) server_negotiation.cc:407] Connection header received
0812 00:58:21.263508 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:21.263514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:21.263611 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:21.263762 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:58:21.264523 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.265434 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.268071 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.268738 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.268841 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:21.269593 (+ 752us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:58:21.269713 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:58:21.269795 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:21.269977 (+ 182us) server_negotiation.cc:299] Negotiation successful
0812 00:58:21.270054 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 00:58:21.270911 26282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:21.261728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:44581 (local address 127.21.201.66:51573)
0812 00:58:21.262759 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:21.262789 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:21.263003 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:21.263780 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:21.263786 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:21.263803 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:58:21.264342 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:21.264356 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.265595 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.265601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:21.267895 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:21.267912 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.269066 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.269072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:21.269250 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:21.269350 (+ 100us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:58:21.269825 (+ 475us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:58:21.269839 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 00:58:21.269937 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 00:58:21.270124 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":696,"threads_started":1}
I20260812 00:58:21.325363 25285 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:58:21.325721 25285 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:58:21.330677 25285 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06: Sending DeleteTablet for 3 replicas of tablet 35a33836aedf45f9a1b6b84f9eb9125f
I20260812 00:58:21.332227 25774 tablet_service.cc:1514] Processing DeleteTablet for tablet 35a33836aedf45f9a1b6b84f9eb9125f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:58:21 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:34662
I20260812 00:58:21.332239 25540 tablet_service.cc:1514] Processing DeleteTablet for tablet 35a33836aedf45f9a1b6b84f9eb9125f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:58:21 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:58774
I20260812 00:58:21.333778 26044 tablet_service.cc:1514] Processing DeleteTablet for tablet 35a33836aedf45f9a1b6b84f9eb9125f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:58:21 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:36938
I20260812 00:58:21.344449 25285 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:58:21.345741 25285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:58:21.349840 26289 tablet_replica.cc:330] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28: stopping tablet replica
I20260812 00:58:21.350610 26289 raft_consensus.cc:2229] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:58:21.351471 26289 raft_consensus.cc:2258] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:58:21.358139 25540 tablet_service.cc:1467] Processing CreateTablet for tablet 99bc2c5e87ee431da6ef74c635a3e77b (DEFAULT_TABLE table=test-table [id=e47dedecb529467bb9d78ef765c33d76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:21.358493 26043 tablet_service.cc:1467] Processing CreateTablet for tablet 99bc2c5e87ee431da6ef74c635a3e77b (DEFAULT_TABLE table=test-table [id=e47dedecb529467bb9d78ef765c33d76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:21.359386 26043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99bc2c5e87ee431da6ef74c635a3e77b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:21.360217 25540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99bc2c5e87ee431da6ef74c635a3e77b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:21.360173 25772 tablet_service.cc:1467] Processing CreateTablet for tablet 99bc2c5e87ee431da6ef74c635a3e77b (DEFAULT_TABLE table=test-table [id=e47dedecb529467bb9d78ef765c33d76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:21.360773 25772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99bc2c5e87ee431da6ef74c635a3e77b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:21.366737 26205 tablet_bootstrap.cc:492] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583: Bootstrap starting.
I20260812 00:58:21.368690 26205 tablet_bootstrap.cc:654] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:21.368885 26201 tablet_bootstrap.cc:492] T 99bc2c5e87ee431da6ef74c635a3e77b P ad6137c90eb04e17bccecb4fa4977244: Bootstrap starting.
I20260812 00:58:21.370754 26201 tablet_bootstrap.cc:654] T 99bc2c5e87ee431da6ef74c635a3e77b P ad6137c90eb04e17bccecb4fa4977244: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:21.372076 26289 ts_tablet_manager.cc:1905] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:58:21.372541 26204 tablet_bootstrap.cc:492] T 99bc2c5e87ee431da6ef74c635a3e77b P 0dbc61bb753e46a9be28f8e789ad8b28: Bootstrap starting.
I20260812 00:58:21.374903 26204 tablet_bootstrap.cc:654] T 99bc2c5e87ee431da6ef74c635a3e77b P 0dbc61bb753e46a9be28f8e789ad8b28: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:21.380089 26292 tablet_replica.cc:330] T 35a33836aedf45f9a1b6b84f9eb9125f P 5848e5a597064322aa7c5b615f34e583: stopping tablet replica
I20260812 00:58:21.381003 26292 raft_consensus.cc:2229] T 35a33836aedf45f9a1b6b84f9eb9125f P 5848e5a597064322aa7c5b615f34e583 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:58:21.381739 26290 tablet_replica.cc:330] T 35a33836aedf45f9a1b6b84f9eb9125f P ad6137c90eb04e17bccecb4fa4977244: stopping tablet replica
I20260812 00:58:21.383760 26204 tablet_bootstrap.cc:492] T 99bc2c5e87ee431da6ef74c635a3e77b P 0dbc61bb753e46a9be28f8e789ad8b28: No bootstrap required, opened a new log
I20260812 00:58:21.384014 26204 ts_tablet_manager.cc:1397] T 99bc2c5e87ee431da6ef74c635a3e77b P 0dbc61bb753e46a9be28f8e789ad8b28: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260812 00:58:21.385048 26204 raft_consensus.cc:348] T 99bc2c5e87ee431da6ef74c635a3e77b P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:21.385367 26204 raft_consensus.cc:374] T 99bc2c5e87ee431da6ef74c635a3e77b P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:21.385461 26204 raft_consensus.cc:729] T 99bc2c5e87ee431da6ef74c635a3e77b P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dbc61bb753e46a9be28f8e789ad8b28, State: Initialized, Role: FOLLOWER
I20260812 00:58:21.385761 26204 consensus_queue.cc:260] T 99bc2c5e87ee431da6ef74c635a3e77b P 0dbc61bb753e46a9be28f8e789ad8b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:21.386445 26204 ts_tablet_manager.cc:1428] T 99bc2c5e87ee431da6ef74c635a3e77b P 0dbc61bb753e46a9be28f8e789ad8b28: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:58:21.385973 26292 raft_consensus.cc:2258] T 35a33836aedf45f9a1b6b84f9eb9125f P 5848e5a597064322aa7c5b615f34e583 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:58:21.386741 26290 raft_consensus.cc:2229] T 35a33836aedf45f9a1b6b84f9eb9125f P ad6137c90eb04e17bccecb4fa4977244 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:58:21.387480 26290 raft_consensus.cc:2258] T 35a33836aedf45f9a1b6b84f9eb9125f P ad6137c90eb04e17bccecb4fa4977244 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:58:21.389631 26292 ts_tablet_manager.cc:1905] T 35a33836aedf45f9a1b6b84f9eb9125f P 5848e5a597064322aa7c5b615f34e583: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:58:21.396838 26201 tablet_bootstrap.cc:492] T 99bc2c5e87ee431da6ef74c635a3e77b P ad6137c90eb04e17bccecb4fa4977244: No bootstrap required, opened a new log
I20260812 00:58:21.397096 26201 ts_tablet_manager.cc:1397] T 99bc2c5e87ee431da6ef74c635a3e77b P ad6137c90eb04e17bccecb4fa4977244: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.000s
I20260812 00:58:21.398207 26201 raft_consensus.cc:348] T 99bc2c5e87ee431da6ef74c635a3e77b P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:21.398569 26201 raft_consensus.cc:374] T 99bc2c5e87ee431da6ef74c635a3e77b P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:21.398677 26201 raft_consensus.cc:729] T 99bc2c5e87ee431da6ef74c635a3e77b P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6137c90eb04e17bccecb4fa4977244, State: Initialized, Role: FOLLOWER
I20260812 00:58:21.399035 26201 consensus_queue.cc:260] T 99bc2c5e87ee431da6ef74c635a3e77b P ad6137c90eb04e17bccecb4fa4977244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:21.400527 26290 ts_tablet_manager.cc:1905] T 35a33836aedf45f9a1b6b84f9eb9125f P ad6137c90eb04e17bccecb4fa4977244: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:58:21.397172 26205 tablet_bootstrap.cc:492] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583: No bootstrap required, opened a new log
I20260812 00:58:21.400957 26205 ts_tablet_manager.cc:1397] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.008s
I20260812 00:58:21.401896 26205 raft_consensus.cc:348] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:21.402179 26205 raft_consensus.cc:374] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:21.402269 26205 raft_consensus.cc:729] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5848e5a597064322aa7c5b615f34e583, State: Initialized, Role: FOLLOWER
I20260812 00:58:21.402550 26205 consensus_queue.cc:260] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:21.403188 26205 ts_tablet_manager.cc:1428] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 00:58:21.404266 26201 ts_tablet_manager.cc:1428] T 99bc2c5e87ee431da6ef74c635a3e77b P ad6137c90eb04e17bccecb4fa4977244: Time spent starting tablet: real 0.007s user 0.003s sys 0.001s
I20260812 00:58:21.414703 26290 ts_tablet_manager.cc:1918] T 35a33836aedf45f9a1b6b84f9eb9125f P ad6137c90eb04e17bccecb4fa4977244: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:58:21.415076 26290 log.cc:1196] T 35a33836aedf45f9a1b6b84f9eb9125f P ad6137c90eb04e17bccecb4fa4977244: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/35a33836aedf45f9a1b6b84f9eb9125f
I20260812 00:58:21.415894 26290 ts_tablet_manager.cc:1939] T 35a33836aedf45f9a1b6b84f9eb9125f P ad6137c90eb04e17bccecb4fa4977244: Deleting consensus metadata
I20260812 00:58:21.418956 25263 catalog_manager.cc:4841] TS ad6137c90eb04e17bccecb4fa4977244 (127.21.201.66:32843): tablet 35a33836aedf45f9a1b6b84f9eb9125f (table test-table [id=392776d93a4c470498eeed8684fc1c33]) successfully deleted
I20260812 00:58:21.421530 26292 ts_tablet_manager.cc:1918] T 35a33836aedf45f9a1b6b84f9eb9125f P 5848e5a597064322aa7c5b615f34e583: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:58:21.421897 26292 log.cc:1196] T 35a33836aedf45f9a1b6b84f9eb9125f P 5848e5a597064322aa7c5b615f34e583: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/35a33836aedf45f9a1b6b84f9eb9125f
I20260812 00:58:21.422626 26292 ts_tablet_manager.cc:1939] T 35a33836aedf45f9a1b6b84f9eb9125f P 5848e5a597064322aa7c5b615f34e583: Deleting consensus metadata
I20260812 00:58:21.424587 26289 ts_tablet_manager.cc:1918] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:58:21.424803 26289 log.cc:1196] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-0/wal/wals/35a33836aedf45f9a1b6b84f9eb9125f
I20260812 00:58:21.425390 26289 ts_tablet_manager.cc:1939] T 35a33836aedf45f9a1b6b84f9eb9125f P 0dbc61bb753e46a9be28f8e789ad8b28: Deleting consensus metadata
I20260812 00:58:21.426148 25262 catalog_manager.cc:4841] TS 5848e5a597064322aa7c5b615f34e583 (127.21.201.67:40315): tablet 35a33836aedf45f9a1b6b84f9eb9125f (table test-table [id=392776d93a4c470498eeed8684fc1c33]) successfully deleted
I20260812 00:58:21.427613 25258 catalog_manager.cc:4841] TS 0dbc61bb753e46a9be28f8e789ad8b28 (127.21.201.65:44581): tablet 35a33836aedf45f9a1b6b84f9eb9125f (table test-table [id=392776d93a4c470498eeed8684fc1c33]) successfully deleted
I20260812 00:58:21.633762 26216 raft_consensus.cc:482] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:58:21.634156 26216 raft_consensus.cc:504] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:21.636734 26216 leader_election.cc:288] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dbc61bb753e46a9be28f8e789ad8b28 (127.21.201.65:44581), ad6137c90eb04e17bccecb4fa4977244 (127.21.201.66:32843)
I20260812 00:58:21.658609 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:21.637284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:43077 (local address 127.21.201.65:44581)
0812 00:58:21.637485 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:21.637491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:21.649375 (+ 11884us) server_negotiation.cc:407] Connection header received
0812 00:58:21.650011 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:21.650020 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:21.650118 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:21.650194 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:58:21.651263 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.652603 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.655468 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.657090 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.657298 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:21.657399 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:21.658045 (+ 646us) server_negotiation.cc:299] Negotiation successful
0812 00:58:21.658222 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 00:58:21.660161 26300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:21.637051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:44581 (local address 127.21.201.67:43077)
0812 00:58:21.649173 (+ 12122us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:21.649207 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:21.649790 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:21.650452 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:21.650457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:21.650485 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:58:21.651041 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:21.651059 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.652816 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.652822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:21.655131 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:21.655152 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.657605 (+ 2453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.657612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:21.657825 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:21.657861 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 00:58:21.658950 (+ 1089us) client_negotiation.cc:240] Negotiation successful
0812 00:58:21.659722 (+ 772us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11940,"thread_start_us":12351,"threads_started":1}
I20260812 00:58:21.660980 25566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bc2c5e87ee431da6ef74c635a3e77b" candidate_uuid: "5848e5a597064322aa7c5b615f34e583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" is_pre_election: true
I20260812 00:58:21.661298 25566 raft_consensus.cc:2454] T 99bc2c5e87ee431da6ef74c635a3e77b P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5848e5a597064322aa7c5b615f34e583 in term 0.
I20260812 00:58:21.661640 26198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:21.638256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:48131 (local address 127.21.201.66:32843)
0812 00:58:21.638469 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:21.638475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:21.650947 (+ 12472us) server_negotiation.cc:407] Connection header received
0812 00:58:21.651569 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:21.651576 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:21.651672 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:21.651773 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:58:21.653006 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.654480 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.657709 (+ 3229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.659359 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.659561 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:21.659687 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:21.661023 (+ 1336us) server_negotiation.cc:299] Negotiation successful
0812 00:58:21.661236 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 00:58:21.661919 25939 leader_election.cc:302] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dbc61bb753e46a9be28f8e789ad8b28, 5848e5a597064322aa7c5b615f34e583; no voters:
I20260812 00:58:21.662531 26216 raft_consensus.cc:2781] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:58:21.662748 26216 raft_consensus.cc:482] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:58:21.662868 26216 raft_consensus.cc:3037] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:21.663731 26302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:21.637833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:32843 (local address 127.21.201.67:48131)
0812 00:58:21.650730 (+ 12897us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:21.650767 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:21.651126 (+ 359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:21.652004 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:21.652010 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:21.652058 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:58:21.652706 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:21.652731 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.654948 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.654954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:21.657417 (+ 2463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:21.657441 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.660503 (+ 3062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.660516 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:21.660763 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:21.660812 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 00:58:21.662296 (+ 1484us) client_negotiation.cc:240] Negotiation successful
0812 00:58:21.663266 (+ 970us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12628,"thread_start_us":12831,"threads_started":1}
I20260812 00:58:21.665040 25815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bc2c5e87ee431da6ef74c635a3e77b" candidate_uuid: "5848e5a597064322aa7c5b615f34e583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6137c90eb04e17bccecb4fa4977244" is_pre_election: true
I20260812 00:58:21.665413 25815 raft_consensus.cc:2454] T 99bc2c5e87ee431da6ef74c635a3e77b P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5848e5a597064322aa7c5b615f34e583 in term 0.
I20260812 00:58:21.666739 26216 raft_consensus.cc:504] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:21.667634 26216 leader_election.cc:288] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583 [CANDIDATE]: Term 1 election: Requested vote from peers 0dbc61bb753e46a9be28f8e789ad8b28 (127.21.201.65:44581), ad6137c90eb04e17bccecb4fa4977244 (127.21.201.66:32843)
I20260812 00:58:21.668635 25566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bc2c5e87ee431da6ef74c635a3e77b" candidate_uuid: "5848e5a597064322aa7c5b615f34e583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dbc61bb753e46a9be28f8e789ad8b28"
I20260812 00:58:21.668960 25566 raft_consensus.cc:3037] T 99bc2c5e87ee431da6ef74c635a3e77b P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:21.668876 25815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bc2c5e87ee431da6ef74c635a3e77b" candidate_uuid: "5848e5a597064322aa7c5b615f34e583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6137c90eb04e17bccecb4fa4977244"
I20260812 00:58:21.669163 25815 raft_consensus.cc:3037] T 99bc2c5e87ee431da6ef74c635a3e77b P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:21.672400 25566 raft_consensus.cc:2454] T 99bc2c5e87ee431da6ef74c635a3e77b P 0dbc61bb753e46a9be28f8e789ad8b28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5848e5a597064322aa7c5b615f34e583 in term 1.
I20260812 00:58:21.673035 25939 leader_election.cc:302] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dbc61bb753e46a9be28f8e789ad8b28, 5848e5a597064322aa7c5b615f34e583; no voters:
I20260812 00:58:21.673513 26216 raft_consensus.cc:2781] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:58:21.674983 25815 raft_consensus.cc:2454] T 99bc2c5e87ee431da6ef74c635a3e77b P ad6137c90eb04e17bccecb4fa4977244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5848e5a597064322aa7c5b615f34e583 in term 1.
I20260812 00:58:21.676319 26216 raft_consensus.cc:686] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583 [term 1 LEADER]: Becoming Leader. State: Replica: 5848e5a597064322aa7c5b615f34e583, State: Running, Role: LEADER
I20260812 00:58:21.676805 26216 consensus_queue.cc:237] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:21.683888 25282 catalog_manager.cc:5495] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5848e5a597064322aa7c5b615f34e583 (127.21.201.67). New cstate: current_term: 1 leader_uuid: "5848e5a597064322aa7c5b615f34e583" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } health_report { overall_health: HEALTHY } } }
I20260812 00:58:21.771270 26314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:21.763469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36944 (local address 127.21.201.67:40315)
0812 00:58:21.764536 (+ 1067us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:21.764541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:21.764564 (+ 23us) server_negotiation.cc:407] Connection header received
0812 00:58:21.764670 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:21.764674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:21.764761 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:21.764968 (+ 207us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:58:21.765653 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.766741 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.768468 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.769038 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.769196 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:21.769717 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:58:21.770456 (+ 739us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:58:21.770631 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:21.770746 (+ 115us) server_negotiation.cc:299] Negotiation successful
0812 00:58:21.770846 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":882,"thread_start_us":673,"threads_started":1}
I20260812 00:58:21.792601 26198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:21.786641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:45903 (local address 127.21.201.66:32843)
0812 00:58:21.786887 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:21.786896 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:21.787244 (+ 348us) server_negotiation.cc:407] Connection header received
0812 00:58:21.787500 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:21.787504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:21.787575 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:21.787682 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:58:21.788296 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.789039 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.790626 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.791063 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.791208 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:21.791728 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:58:21.791933 (+ 205us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:58:21.792037 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:21.792183 (+ 146us) server_negotiation.cc:299] Negotiation successful
0812 00:58:21.792274 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 00:58:21.792833 26317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:21.786394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:32843 (local address 127.21.201.67:45903)
0812 00:58:21.787128 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:21.787144 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:21.787337 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:21.787698 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:21.787702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:21.787714 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:58:21.788119 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:21.788128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.789191 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.789196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:21.790463 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:21.790476 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.791294 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.791300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:21.791443 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:21.791519 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:58:21.792039 (+ 520us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:58:21.792055 (+ 16us) client_negotiation.cc:769] Sending connection context
0812 00:58:21.792158 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 00:58:21.792324 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":347,"threads_started":1}
I20260812 00:58:21.803402 25815 raft_consensus.cc:1261] T 99bc2c5e87ee431da6ef74c635a3e77b P ad6137c90eb04e17bccecb4fa4977244 [term 1 FOLLOWER]: Refusing update from remote peer 5848e5a597064322aa7c5b615f34e583: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:58:21.803586 25565 raft_consensus.cc:1261] T 99bc2c5e87ee431da6ef74c635a3e77b P 0dbc61bb753e46a9be28f8e789ad8b28 [term 1 FOLLOWER]: Refusing update from remote peer 5848e5a597064322aa7c5b615f34e583: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:58:21.804582 26304 consensus_queue.cc:1035] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:58:21.805151 26216 consensus_queue.cc:1035] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:58:21.818732 26258 mvcc.cc:205] Tried to move back new op lower bound from 7317488852180221952 to 7317488851675123712. Current Snapshot: MvccSnapshot[applied={T|T < 7317488852180221952}]
I20260812 00:58:21.855250 26314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:21.848106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:53963 (local address 127.21.201.67:40315)
0812 00:58:21.848472 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:21.848481 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:21.848831 (+ 350us) server_negotiation.cc:407] Connection header received
0812 00:58:21.849143 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:21.849148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:21.849237 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:21.849352 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:58:21.849790 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.850819 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.852998 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.853658 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.853773 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:21.854340 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:58:21.854527 (+ 187us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:58:21.854605 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:21.854792 (+ 187us) server_negotiation.cc:299] Negotiation successful
0812 00:58:21.854869 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260812 00:58:21.855398 26325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:21.847853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:40315 (local address 127.21.201.66:53963)
0812 00:58:21.848717 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:21.848741 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:21.848972 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:21.849368 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:21.849373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:21.849386 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:58:21.849639 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:21.849649 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.851007 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.851012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:21.852824 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:21.852842 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:21.853866 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:21.853874 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:21.854077 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:21.854161 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:58:21.854652 (+ 491us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:58:21.854665 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 00:58:21.854766 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 00:58:21.854944 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":511,"threads_started":1}
W20260812 00:58:21.856775 26259 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 00:58:23.496140 25285 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:58:23.496483 25285 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:58:23.501487 25285 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06: Sending DeleteTablet for 3 replicas of tablet 99bc2c5e87ee431da6ef74c635a3e77b
I20260812 00:58:23.502758 25774 tablet_service.cc:1514] Processing DeleteTablet for tablet 99bc2c5e87ee431da6ef74c635a3e77b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:58:23 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:34662
I20260812 00:58:23.502753 26044 tablet_service.cc:1514] Processing DeleteTablet for tablet 99bc2c5e87ee431da6ef74c635a3e77b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:58:23 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:36938
I20260812 00:58:23.502964 25540 tablet_service.cc:1514] Processing DeleteTablet for tablet 99bc2c5e87ee431da6ef74c635a3e77b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:58:23 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:58774
I20260812 00:58:23.515602 26364 tablet_replica.cc:330] T 99bc2c5e87ee431da6ef74c635a3e77b P ad6137c90eb04e17bccecb4fa4977244: stopping tablet replica
I20260812 00:58:23.516248 26364 raft_consensus.cc:2229] T 99bc2c5e87ee431da6ef74c635a3e77b P ad6137c90eb04e17bccecb4fa4977244 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:58:23.516696 26364 raft_consensus.cc:2258] T 99bc2c5e87ee431da6ef74c635a3e77b P ad6137c90eb04e17bccecb4fa4977244 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:58:23.518501 26364 ts_tablet_manager.cc:1905] T 99bc2c5e87ee431da6ef74c635a3e77b P ad6137c90eb04e17bccecb4fa4977244: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:58:23.520006 26366 tablet_replica.cc:330] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583: stopping tablet replica
I20260812 00:58:23.520314 26366 raft_consensus.cc:2229] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:58:23.521078 26366 raft_consensus.cc:2258] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:58:23.522110 26368 tablet_replica.cc:330] T 99bc2c5e87ee431da6ef74c635a3e77b P 0dbc61bb753e46a9be28f8e789ad8b28: stopping tablet replica
I20260812 00:58:23.522352 26368 raft_consensus.cc:2229] T 99bc2c5e87ee431da6ef74c635a3e77b P 0dbc61bb753e46a9be28f8e789ad8b28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:58:23.522612 26368 raft_consensus.cc:2258] T 99bc2c5e87ee431da6ef74c635a3e77b P 0dbc61bb753e46a9be28f8e789ad8b28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:58:23.523384 26366 ts_tablet_manager.cc:1905] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:58:23.524276 26368 ts_tablet_manager.cc:1905] T 99bc2c5e87ee431da6ef74c635a3e77b P 0dbc61bb753e46a9be28f8e789ad8b28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:58:23.526734 26364 ts_tablet_manager.cc:1918] T 99bc2c5e87ee431da6ef74c635a3e77b P ad6137c90eb04e17bccecb4fa4977244: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 00:58:23.526949 26364 log.cc:1196] T 99bc2c5e87ee431da6ef74c635a3e77b P ad6137c90eb04e17bccecb4fa4977244: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/99bc2c5e87ee431da6ef74c635a3e77b
I20260812 00:58:23.527504 26364 ts_tablet_manager.cc:1939] T 99bc2c5e87ee431da6ef74c635a3e77b P ad6137c90eb04e17bccecb4fa4977244: Deleting consensus metadata
I20260812 00:58:23.529011 25263 catalog_manager.cc:4841] TS ad6137c90eb04e17bccecb4fa4977244 (127.21.201.66:32843): tablet 99bc2c5e87ee431da6ef74c635a3e77b (table test-table [id=e47dedecb529467bb9d78ef765c33d76]) successfully deleted
I20260812 00:58:23.532305 26366 ts_tablet_manager.cc:1918] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 00:58:23.532534 26366 log.cc:1196] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/99bc2c5e87ee431da6ef74c635a3e77b
I20260812 00:58:23.533147 26366 ts_tablet_manager.cc:1939] T 99bc2c5e87ee431da6ef74c635a3e77b P 5848e5a597064322aa7c5b615f34e583: Deleting consensus metadata
I20260812 00:58:23.533387 26368 ts_tablet_manager.cc:1918] T 99bc2c5e87ee431da6ef74c635a3e77b P 0dbc61bb753e46a9be28f8e789ad8b28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 00:58:23.533555 26368 log.cc:1196] T 99bc2c5e87ee431da6ef74c635a3e77b P 0dbc61bb753e46a9be28f8e789ad8b28: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-0/wal/wals/99bc2c5e87ee431da6ef74c635a3e77b
I20260812 00:58:23.534029 26368 ts_tablet_manager.cc:1939] T 99bc2c5e87ee431da6ef74c635a3e77b P 0dbc61bb753e46a9be28f8e789ad8b28: Deleting consensus metadata
I20260812 00:58:23.534778 25262 catalog_manager.cc:4841] TS 5848e5a597064322aa7c5b615f34e583 (127.21.201.67:40315): tablet 99bc2c5e87ee431da6ef74c635a3e77b (table test-table [id=e47dedecb529467bb9d78ef765c33d76]) successfully deleted
I20260812 00:58:23.535259 25258 catalog_manager.cc:4841] TS 0dbc61bb753e46a9be28f8e789ad8b28 (127.21.201.65:44581): tablet 99bc2c5e87ee431da6ef74c635a3e77b (table test-table [id=e47dedecb529467bb9d78ef765c33d76]) successfully deleted
Aug 12 00:58:23 dist-test-slave-7f01 krb5kdc[25213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496303, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 00:58:23 dist-test-slave-7f01 krb5kdc[25213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496303, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:23.592006 26379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:23.567856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58236 (local address 127.21.201.126:38635)
0812 00:58:23.568556 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:23.568561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:23.568590 (+ 29us) server_negotiation.cc:407] Connection header received
0812 00:58:23.568863 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:23.568869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:23.568982 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:23.569135 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:23.570963 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:23.572240 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:23.573814 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:23.574326 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:23.574491 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:23.584012 (+ 9521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:23.584021 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:23.584026 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:23.584094 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:23.588352 (+ 4258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:23.589351 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:23.589357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:23.589359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:23.589586 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:23.590073 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:23.590077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:23.590080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:23.590469 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:23.590720 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:23.591235 (+ 515us) server_negotiation.cc:299] Negotiation successful
0812 00:58:23.591569 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":346,"threads_started":1}
I20260812 00:58:23.596817 25285 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:58:23.597780 25285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:58:23.607308 26044 tablet_service.cc:1467] Processing CreateTablet for tablet bcb975401b044884b837e376f408b02a (DEFAULT_TABLE table=test-table [id=a1804f964b824241bc3ffcf14dfc93ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:23.607298 25540 tablet_service.cc:1467] Processing CreateTablet for tablet bcb975401b044884b837e376f408b02a (DEFAULT_TABLE table=test-table [id=a1804f964b824241bc3ffcf14dfc93ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:23.607268 25774 tablet_service.cc:1467] Processing CreateTablet for tablet bcb975401b044884b837e376f408b02a (DEFAULT_TABLE table=test-table [id=a1804f964b824241bc3ffcf14dfc93ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:23.607849 26044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcb975401b044884b837e376f408b02a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:23.607848 25774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcb975401b044884b837e376f408b02a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:23.607867 25540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcb975401b044884b837e376f408b02a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:23.614707 26380 tablet_bootstrap.cc:492] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244: Bootstrap starting.
I20260812 00:58:23.615459 26383 tablet_bootstrap.cc:492] T bcb975401b044884b837e376f408b02a P 5848e5a597064322aa7c5b615f34e583: Bootstrap starting.
I20260812 00:58:23.617048 26380 tablet_bootstrap.cc:654] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:23.617354 26383 tablet_bootstrap.cc:654] T bcb975401b044884b837e376f408b02a P 5848e5a597064322aa7c5b615f34e583: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:23.617955 26382 tablet_bootstrap.cc:492] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28: Bootstrap starting.
I20260812 00:58:23.620142 26383 tablet_bootstrap.cc:492] T bcb975401b044884b837e376f408b02a P 5848e5a597064322aa7c5b615f34e583: No bootstrap required, opened a new log
I20260812 00:58:23.620249 26382 tablet_bootstrap.cc:654] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:23.620317 26383 ts_tablet_manager.cc:1397] T bcb975401b044884b837e376f408b02a P 5848e5a597064322aa7c5b615f34e583: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260812 00:58:23.620932 26380 tablet_bootstrap.cc:492] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244: No bootstrap required, opened a new log
I20260812 00:58:23.621104 26380 ts_tablet_manager.cc:1397] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 00:58:23.621079 26383 raft_consensus.cc:348] T bcb975401b044884b837e376f408b02a P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:23.621392 26383 raft_consensus.cc:374] T bcb975401b044884b837e376f408b02a P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:23.621502 26383 raft_consensus.cc:729] T bcb975401b044884b837e376f408b02a P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5848e5a597064322aa7c5b615f34e583, State: Initialized, Role: FOLLOWER
I20260812 00:58:23.621796 26383 consensus_queue.cc:260] T bcb975401b044884b837e376f408b02a P 5848e5a597064322aa7c5b615f34e583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:23.622143 26380 raft_consensus.cc:348] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:23.622411 26380 raft_consensus.cc:374] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:23.622468 26383 ts_tablet_manager.cc:1428] T bcb975401b044884b837e376f408b02a P 5848e5a597064322aa7c5b615f34e583: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:58:23.622524 26380 raft_consensus.cc:729] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6137c90eb04e17bccecb4fa4977244, State: Initialized, Role: FOLLOWER
I20260812 00:58:23.622840 26380 consensus_queue.cc:260] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:23.623399 26382 tablet_bootstrap.cc:492] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28: No bootstrap required, opened a new log
I20260812 00:58:23.623440 26380 ts_tablet_manager.cc:1428] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:58:23.623639 26382 ts_tablet_manager.cc:1397] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 00:58:23.624600 26382 raft_consensus.cc:348] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:23.624847 26382 raft_consensus.cc:374] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:23.624959 26382 raft_consensus.cc:729] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dbc61bb753e46a9be28f8e789ad8b28, State: Initialized, Role: FOLLOWER
I20260812 00:58:23.625242 26382 consensus_queue.cc:260] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:23.626256 26382 ts_tablet_manager.cc:1428] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:58:23.988411 26392 raft_consensus.cc:482] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:58:23.988552 26266 raft_consensus.cc:482] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:58:23.988739 26392 raft_consensus.cc:504] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:23.988862 26266 raft_consensus.cc:504] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:23.989832 26266 leader_election.cc:288] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dbc61bb753e46a9be28f8e789ad8b28 (127.21.201.65:44581), 5848e5a597064322aa7c5b615f34e583 (127.21.201.67:40315)
I20260812 00:58:23.989888 26392 leader_election.cc:288] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad6137c90eb04e17bccecb4fa4977244 (127.21.201.66:32843), 5848e5a597064322aa7c5b615f34e583 (127.21.201.67:40315)
I20260812 00:58:23.990746 25565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb975401b044884b837e376f408b02a" candidate_uuid: "ad6137c90eb04e17bccecb4fa4977244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" is_pre_election: true
I20260812 00:58:23.990833 26084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb975401b044884b837e376f408b02a" candidate_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5848e5a597064322aa7c5b615f34e583" is_pre_election: true
I20260812 00:58:23.991008 26085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb975401b044884b837e376f408b02a" candidate_uuid: "ad6137c90eb04e17bccecb4fa4977244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5848e5a597064322aa7c5b615f34e583" is_pre_election: true
I20260812 00:58:23.991135 25565 raft_consensus.cc:2454] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad6137c90eb04e17bccecb4fa4977244 in term 0.
I20260812 00:58:23.991309 26084 raft_consensus.cc:2454] T bcb975401b044884b837e376f408b02a P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dbc61bb753e46a9be28f8e789ad8b28 in term 0.
I20260812 00:58:23.991291 25814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb975401b044884b837e376f408b02a" candidate_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6137c90eb04e17bccecb4fa4977244" is_pre_election: true
I20260812 00:58:23.991564 25814 raft_consensus.cc:2454] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dbc61bb753e46a9be28f8e789ad8b28 in term 0.
I20260812 00:58:23.992005 25450 leader_election.cc:302] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dbc61bb753e46a9be28f8e789ad8b28, 5848e5a597064322aa7c5b615f34e583; no voters:
I20260812 00:58:23.992201 25696 leader_election.cc:302] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dbc61bb753e46a9be28f8e789ad8b28, ad6137c90eb04e17bccecb4fa4977244; no voters:
I20260812 00:58:23.992405 26392 raft_consensus.cc:2781] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:58:23.992542 26392 raft_consensus.cc:482] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:58:23.992645 26392 raft_consensus.cc:3037] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:23.992853 26266 raft_consensus.cc:2781] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:58:23.993067 26266 raft_consensus.cc:482] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:58:23.993206 26266 raft_consensus.cc:3037] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:23.995052 26392 raft_consensus.cc:504] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:23.995324 26266 raft_consensus.cc:504] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:23.995817 26392 leader_election.cc:288] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28 [CANDIDATE]: Term 1 election: Requested vote from peers ad6137c90eb04e17bccecb4fa4977244 (127.21.201.66:32843), 5848e5a597064322aa7c5b615f34e583 (127.21.201.67:40315)
I20260812 00:58:23.996124 26266 leader_election.cc:288] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244 [CANDIDATE]: Term 1 election: Requested vote from peers 0dbc61bb753e46a9be28f8e789ad8b28 (127.21.201.65:44581), 5848e5a597064322aa7c5b615f34e583 (127.21.201.67:40315)
I20260812 00:58:23.996546 26084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb975401b044884b837e376f408b02a" candidate_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5848e5a597064322aa7c5b615f34e583"
I20260812 00:58:23.996831 26084 raft_consensus.cc:3037] T bcb975401b044884b837e376f408b02a P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:23.996829 25814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb975401b044884b837e376f408b02a" candidate_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6137c90eb04e17bccecb4fa4977244"
I20260812 00:58:23.996872 26085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb975401b044884b837e376f408b02a" candidate_uuid: "ad6137c90eb04e17bccecb4fa4977244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5848e5a597064322aa7c5b615f34e583"
I20260812 00:58:23.996865 25565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb975401b044884b837e376f408b02a" candidate_uuid: "ad6137c90eb04e17bccecb4fa4977244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dbc61bb753e46a9be28f8e789ad8b28"
I20260812 00:58:23.997166 25814 raft_consensus.cc:2379] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0dbc61bb753e46a9be28f8e789ad8b28 in current term 1: Already voted for candidate ad6137c90eb04e17bccecb4fa4977244 in this term.
I20260812 00:58:23.997200 25565 raft_consensus.cc:2379] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ad6137c90eb04e17bccecb4fa4977244 in current term 1: Already voted for candidate 0dbc61bb753e46a9be28f8e789ad8b28 in this term.
I20260812 00:58:23.997753 25696 leader_election.cc:302] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad6137c90eb04e17bccecb4fa4977244; no voters: 0dbc61bb753e46a9be28f8e789ad8b28, 5848e5a597064322aa7c5b615f34e583
I20260812 00:58:23.998200 26266 raft_consensus.cc:2726] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260812 00:58:23.999320 26084 raft_consensus.cc:2454] T bcb975401b044884b837e376f408b02a P 5848e5a597064322aa7c5b615f34e583 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dbc61bb753e46a9be28f8e789ad8b28 in term 1.
I20260812 00:58:23.999857 25450 leader_election.cc:302] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0dbc61bb753e46a9be28f8e789ad8b28, 5848e5a597064322aa7c5b615f34e583; no voters: ad6137c90eb04e17bccecb4fa4977244
I20260812 00:58:24.000186 26392 raft_consensus.cc:2781] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:58:24.000612 26392 raft_consensus.cc:686] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28 [term 1 LEADER]: Becoming Leader. State: Replica: 0dbc61bb753e46a9be28f8e789ad8b28, State: Running, Role: LEADER
I20260812 00:58:24.000944 26392 consensus_queue.cc:237] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:24.004145 25285 catalog_manager.cc:5495] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dbc61bb753e46a9be28f8e789ad8b28 (127.21.201.65). New cstate: current_term: 1 leader_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } health_report { overall_health: UNKNOWN } } }
I20260812 00:58:24.100453 26419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:24.093183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58790 (local address 127.21.201.65:44581)
0812 00:58:24.094130 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:24.094134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:24.094156 (+ 22us) server_negotiation.cc:407] Connection header received
0812 00:58:24.094219 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:24.094222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:24.094297 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:24.094387 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:58:24.094978 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:24.095815 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:24.098065 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:24.098598 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:24.098719 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:24.099311 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:58:24.099436 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:58:24.099509 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:24.099830 (+ 321us) server_negotiation.cc:299] Negotiation successful
0812 00:58:24.099921 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":821,"thread_start_us":675,"threads_started":1}
I20260812 00:58:24.114006 26084 raft_consensus.cc:1261] T bcb975401b044884b837e376f408b02a P 5848e5a597064322aa7c5b615f34e583 [term 1 FOLLOWER]: Refusing update from remote peer 0dbc61bb753e46a9be28f8e789ad8b28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:58:24.114254 25814 raft_consensus.cc:1261] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244 [term 1 FOLLOWER]: Refusing update from remote peer 0dbc61bb753e46a9be28f8e789ad8b28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:58:24.114650 26394 consensus_queue.cc:1035] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:58:24.115027 26392 consensus_queue.cc:1035] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:58:24.168416 25282 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:58:24.168673 25282 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:58:24.171691 25282 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06: Sending DeleteTablet for 3 replicas of tablet bcb975401b044884b837e376f408b02a
I20260812 00:58:24.172595 25540 tablet_service.cc:1514] Processing DeleteTablet for tablet bcb975401b044884b837e376f408b02a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:58:24 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:58774
I20260812 00:58:24.172703 25774 tablet_service.cc:1514] Processing DeleteTablet for tablet bcb975401b044884b837e376f408b02a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:58:24 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:34662
I20260812 00:58:24.172762 26044 tablet_service.cc:1514] Processing DeleteTablet for tablet bcb975401b044884b837e376f408b02a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:58:24 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:36938
I20260812 00:58:24.173489 26429 tablet_replica.cc:330] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28: stopping tablet replica
I20260812 00:58:24.173827 26429 raft_consensus.cc:2229] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:58:24.174374 26429 raft_consensus.cc:2258] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:58:24.174496 25282 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:58:24.175429 25282 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:58:24.176141 26429 ts_tablet_manager.cc:1905] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:58:24.184510 26429 ts_tablet_manager.cc:1918] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:58:24.184448 25772 tablet_service.cc:1467] Processing CreateTablet for tablet ad75be9a79b94d21805860412dc4495d (DEFAULT_TABLE table=test-table [id=2f22e94d18704c72abfa7bd46f7a0755]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:24.184540 26043 tablet_service.cc:1467] Processing CreateTablet for tablet ad75be9a79b94d21805860412dc4495d (DEFAULT_TABLE table=test-table [id=2f22e94d18704c72abfa7bd46f7a0755]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:24.185052 26043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad75be9a79b94d21805860412dc4495d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:24.185110 25772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad75be9a79b94d21805860412dc4495d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:24.185359 26430 tablet_replica.cc:330] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244: stopping tablet replica
I20260812 00:58:24.185635 26430 raft_consensus.cc:2229] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:58:24.185966 26430 raft_consensus.cc:2258] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:58:24.186318 26429 log.cc:1196] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-0/wal/wals/bcb975401b044884b837e376f408b02a
I20260812 00:58:24.186981 26429 ts_tablet_manager.cc:1939] T bcb975401b044884b837e376f408b02a P 0dbc61bb753e46a9be28f8e789ad8b28: Deleting consensus metadata
I20260812 00:58:24.187250 26430 ts_tablet_manager.cc:1905] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:58:24.188356 25258 catalog_manager.cc:4841] TS 0dbc61bb753e46a9be28f8e789ad8b28 (127.21.201.65:44581): tablet bcb975401b044884b837e376f408b02a (table test-table [id=a1804f964b824241bc3ffcf14dfc93ae]) successfully deleted
I20260812 00:58:24.188436 25540 tablet_service.cc:1467] Processing CreateTablet for tablet ad75be9a79b94d21805860412dc4495d (DEFAULT_TABLE table=test-table [id=2f22e94d18704c72abfa7bd46f7a0755]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:24.188843 25540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad75be9a79b94d21805860412dc4495d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:24.194511 26430 ts_tablet_manager.cc:1918] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:58:24.194720 26430 log.cc:1196] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/bcb975401b044884b837e376f408b02a
I20260812 00:58:24.195266 26430 ts_tablet_manager.cc:1939] T bcb975401b044884b837e376f408b02a P ad6137c90eb04e17bccecb4fa4977244: Deleting consensus metadata
I20260812 00:58:24.196097 26431 tablet_replica.cc:330] T bcb975401b044884b837e376f408b02a P 5848e5a597064322aa7c5b615f34e583: stopping tablet replica
I20260812 00:58:24.196735 26431 raft_consensus.cc:2229] T bcb975401b044884b837e376f408b02a P 5848e5a597064322aa7c5b615f34e583 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:58:24.197062 26431 raft_consensus.cc:2258] T bcb975401b044884b837e376f408b02a P 5848e5a597064322aa7c5b615f34e583 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:58:24.198212 26431 ts_tablet_manager.cc:1905] T bcb975401b044884b837e376f408b02a P 5848e5a597064322aa7c5b615f34e583: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:58:24.200182 26432 tablet_bootstrap.cc:492] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244: Bootstrap starting.
I20260812 00:58:24.200598 26434 tablet_bootstrap.cc:492] T ad75be9a79b94d21805860412dc4495d P 5848e5a597064322aa7c5b615f34e583: Bootstrap starting.
I20260812 00:58:24.202750 25263 catalog_manager.cc:4841] TS ad6137c90eb04e17bccecb4fa4977244 (127.21.201.66:32843): tablet bcb975401b044884b837e376f408b02a (table test-table [id=a1804f964b824241bc3ffcf14dfc93ae]) successfully deleted
I20260812 00:58:24.203239 26434 tablet_bootstrap.cc:654] T ad75be9a79b94d21805860412dc4495d P 5848e5a597064322aa7c5b615f34e583: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:24.204519 26432 tablet_bootstrap.cc:654] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:24.205451 26436 tablet_bootstrap.cc:492] T ad75be9a79b94d21805860412dc4495d P 0dbc61bb753e46a9be28f8e789ad8b28: Bootstrap starting.
I20260812 00:58:24.206913 26431 ts_tablet_manager.cc:1918] T bcb975401b044884b837e376f408b02a P 5848e5a597064322aa7c5b615f34e583: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:58:24.207086 26431 log.cc:1196] T bcb975401b044884b837e376f408b02a P 5848e5a597064322aa7c5b615f34e583: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/bcb975401b044884b837e376f408b02a
I20260812 00:58:24.207605 26431 ts_tablet_manager.cc:1939] T bcb975401b044884b837e376f408b02a P 5848e5a597064322aa7c5b615f34e583: Deleting consensus metadata
I20260812 00:58:24.207875 26436 tablet_bootstrap.cc:654] T ad75be9a79b94d21805860412dc4495d P 0dbc61bb753e46a9be28f8e789ad8b28: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:24.209095 25262 catalog_manager.cc:4841] TS 5848e5a597064322aa7c5b615f34e583 (127.21.201.67:40315): tablet bcb975401b044884b837e376f408b02a (table test-table [id=a1804f964b824241bc3ffcf14dfc93ae]) successfully deleted
I20260812 00:58:24.210971 26436 tablet_bootstrap.cc:492] T ad75be9a79b94d21805860412dc4495d P 0dbc61bb753e46a9be28f8e789ad8b28: No bootstrap required, opened a new log
I20260812 00:58:24.211134 26436 ts_tablet_manager.cc:1397] T ad75be9a79b94d21805860412dc4495d P 0dbc61bb753e46a9be28f8e789ad8b28: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260812 00:58:24.211443 26432 tablet_bootstrap.cc:492] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244: No bootstrap required, opened a new log
I20260812 00:58:24.211608 26432 ts_tablet_manager.cc:1397] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20260812 00:58:24.212107 26434 tablet_bootstrap.cc:492] T ad75be9a79b94d21805860412dc4495d P 5848e5a597064322aa7c5b615f34e583: No bootstrap required, opened a new log
I20260812 00:58:24.212272 26434 ts_tablet_manager.cc:1397] T ad75be9a79b94d21805860412dc4495d P 5848e5a597064322aa7c5b615f34e583: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260812 00:58:24.212368 26436 raft_consensus.cc:348] T ad75be9a79b94d21805860412dc4495d P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:24.212687 26436 raft_consensus.cc:374] T ad75be9a79b94d21805860412dc4495d P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:24.212788 26436 raft_consensus.cc:729] T ad75be9a79b94d21805860412dc4495d P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dbc61bb753e46a9be28f8e789ad8b28, State: Initialized, Role: FOLLOWER
I20260812 00:58:24.212877 26432 raft_consensus.cc:348] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:24.213172 26432 raft_consensus.cc:374] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:24.213277 26432 raft_consensus.cc:729] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6137c90eb04e17bccecb4fa4977244, State: Initialized, Role: FOLLOWER
I20260812 00:58:24.213127 26436 consensus_queue.cc:260] T ad75be9a79b94d21805860412dc4495d P 0dbc61bb753e46a9be28f8e789ad8b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:24.213194 26434 raft_consensus.cc:348] T ad75be9a79b94d21805860412dc4495d P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:24.213501 26434 raft_consensus.cc:374] T ad75be9a79b94d21805860412dc4495d P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:24.213653 26434 raft_consensus.cc:729] T ad75be9a79b94d21805860412dc4495d P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5848e5a597064322aa7c5b615f34e583, State: Initialized, Role: FOLLOWER
I20260812 00:58:24.213579 26432 consensus_queue.cc:260] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:24.213868 26436 ts_tablet_manager.cc:1428] T ad75be9a79b94d21805860412dc4495d P 0dbc61bb753e46a9be28f8e789ad8b28: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:58:24.213878 26434 consensus_queue.cc:260] T ad75be9a79b94d21805860412dc4495d P 5848e5a597064322aa7c5b615f34e583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:24.214550 26434 ts_tablet_manager.cc:1428] T ad75be9a79b94d21805860412dc4495d P 5848e5a597064322aa7c5b615f34e583: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:58:24.215175 26432 ts_tablet_manager.cc:1428] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:58:24.284947 26406 raft_consensus.cc:482] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:58:24.285218 26406 raft_consensus.cc:504] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:24.285916 26406 leader_election.cc:288] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dbc61bb753e46a9be28f8e789ad8b28 (127.21.201.65:44581), 5848e5a597064322aa7c5b615f34e583 (127.21.201.67:40315)
I20260812 00:58:24.286628 26084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad75be9a79b94d21805860412dc4495d" candidate_uuid: "ad6137c90eb04e17bccecb4fa4977244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5848e5a597064322aa7c5b615f34e583" is_pre_election: true
I20260812 00:58:24.286609 25565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad75be9a79b94d21805860412dc4495d" candidate_uuid: "ad6137c90eb04e17bccecb4fa4977244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" is_pre_election: true
I20260812 00:58:24.286904 26084 raft_consensus.cc:2454] T ad75be9a79b94d21805860412dc4495d P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad6137c90eb04e17bccecb4fa4977244 in term 0.
I20260812 00:58:24.286943 25565 raft_consensus.cc:2454] T ad75be9a79b94d21805860412dc4495d P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad6137c90eb04e17bccecb4fa4977244 in term 0.
I20260812 00:58:24.287400 25696 leader_election.cc:302] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0dbc61bb753e46a9be28f8e789ad8b28, 5848e5a597064322aa7c5b615f34e583, ad6137c90eb04e17bccecb4fa4977244; no voters:
I20260812 00:58:24.287721 26406 raft_consensus.cc:2781] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:58:24.287957 26406 raft_consensus.cc:482] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:58:24.288079 26406 raft_consensus.cc:3037] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:24.290354 26406 raft_consensus.cc:504] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:24.290995 26406 leader_election.cc:288] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244 [CANDIDATE]: Term 1 election: Requested vote from peers 0dbc61bb753e46a9be28f8e789ad8b28 (127.21.201.65:44581), 5848e5a597064322aa7c5b615f34e583 (127.21.201.67:40315)
I20260812 00:58:24.291690 25565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad75be9a79b94d21805860412dc4495d" candidate_uuid: "ad6137c90eb04e17bccecb4fa4977244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dbc61bb753e46a9be28f8e789ad8b28"
I20260812 00:58:24.291699 26084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad75be9a79b94d21805860412dc4495d" candidate_uuid: "ad6137c90eb04e17bccecb4fa4977244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5848e5a597064322aa7c5b615f34e583"
I20260812 00:58:24.291980 26084 raft_consensus.cc:3037] T ad75be9a79b94d21805860412dc4495d P 5848e5a597064322aa7c5b615f34e583 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:24.292001 25565 raft_consensus.cc:3037] T ad75be9a79b94d21805860412dc4495d P 0dbc61bb753e46a9be28f8e789ad8b28 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:24.295188 26084 raft_consensus.cc:2454] T ad75be9a79b94d21805860412dc4495d P 5848e5a597064322aa7c5b615f34e583 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad6137c90eb04e17bccecb4fa4977244 in term 1.
I20260812 00:58:24.295173 25565 raft_consensus.cc:2454] T ad75be9a79b94d21805860412dc4495d P 0dbc61bb753e46a9be28f8e789ad8b28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad6137c90eb04e17bccecb4fa4977244 in term 1.
I20260812 00:58:24.295732 25699 leader_election.cc:302] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0dbc61bb753e46a9be28f8e789ad8b28, 5848e5a597064322aa7c5b615f34e583, ad6137c90eb04e17bccecb4fa4977244; no voters:
I20260812 00:58:24.296131 26406 raft_consensus.cc:2781] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:58:24.296311 26406 raft_consensus.cc:686] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244 [term 1 LEADER]: Becoming Leader. State: Replica: ad6137c90eb04e17bccecb4fa4977244, State: Running, Role: LEADER
I20260812 00:58:24.296603 26406 consensus_queue.cc:237] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } }
I20260812 00:58:24.299443 25288 catalog_manager.cc:5495] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad6137c90eb04e17bccecb4fa4977244 (127.21.201.66). New cstate: current_term: 1 leader_uuid: "ad6137c90eb04e17bccecb4fa4977244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad6137c90eb04e17bccecb4fa4977244" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 32843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 } health_report { overall_health: UNKNOWN } } }
I20260812 00:58:24.344374 26443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:24.337738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34686 (local address 127.21.201.66:32843)
0812 00:58:24.338533 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:24.338537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:24.338560 (+ 23us) server_negotiation.cc:407] Connection header received
0812 00:58:24.338622 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:24.338625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:24.338696 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:24.338790 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:58:24.339288 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:24.340343 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:24.342203 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:24.342700 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:24.342828 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:24.343512 (+ 684us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:58:24.343646 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:58:24.343726 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:24.343915 (+ 189us) server_negotiation.cc:299] Negotiation successful
0812 00:58:24.344016 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":419,"threads_started":1}
I20260812 00:58:24.366061 26443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:24.357724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:59855 (local address 127.21.201.66:32843)
0812 00:58:24.357977 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:24.357987 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:24.358642 (+ 655us) server_negotiation.cc:407] Connection header received
0812 00:58:24.359011 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:24.359016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:24.359114 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:24.359217 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:58:24.360193 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:24.361046 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:24.363513 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:24.364048 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:24.364211 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:24.364827 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:58:24.365077 (+ 250us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:58:24.365286 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:24.365489 (+ 203us) server_negotiation.cc:299] Negotiation successful
0812 00:58:24.365760 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":87}
I20260812 00:58:24.366300 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:24.357383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:32843 (local address 127.21.201.66:59855)
0812 00:58:24.358437 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:24.358473 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:24.358731 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:24.359315 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:24.359321 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:24.359336 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:58:24.359853 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:24.359867 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:24.361224 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:24.361230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:24.363237 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:24.363255 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:24.364278 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:24.364285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:24.364496 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:24.364549 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:58:24.365296 (+ 747us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:58:24.365310 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 00:58:24.365399 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 00:58:24.365623 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":6,"thread_start_us":596,"threads_started":1}
I20260812 00:58:24.376803 26084 raft_consensus.cc:1261] T ad75be9a79b94d21805860412dc4495d P 5848e5a597064322aa7c5b615f34e583 [term 1 FOLLOWER]: Refusing update from remote peer ad6137c90eb04e17bccecb4fa4977244: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:58:24.376868 25565 raft_consensus.cc:1261] T ad75be9a79b94d21805860412dc4495d P 0dbc61bb753e46a9be28f8e789ad8b28 [term 1 FOLLOWER]: Refusing update from remote peer ad6137c90eb04e17bccecb4fa4977244: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:58:24.377712 26406 consensus_queue.cc:1035] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5848e5a597064322aa7c5b615f34e583" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 40315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:58:24.378010 26266 consensus_queue.cc:1035] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dbc61bb753e46a9be28f8e789ad8b28" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 44581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 00:58:24.405243 26396 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 00:58:26.026206 25285 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:58:26.026619 25285 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:58:26.035629 25285 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aeefe634cbe24087837667b7ba19fe06: Sending DeleteTablet for 3 replicas of tablet ad75be9a79b94d21805860412dc4495d
I20260812 00:58:26.036963 25540 tablet_service.cc:1514] Processing DeleteTablet for tablet ad75be9a79b94d21805860412dc4495d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:58:26 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:58774
I20260812 00:58:26.037058 26043 tablet_service.cc:1514] Processing DeleteTablet for tablet ad75be9a79b94d21805860412dc4495d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:58:26 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:36938
I20260812 00:58:26.037082 25772 tablet_service.cc:1514] Processing DeleteTablet for tablet ad75be9a79b94d21805860412dc4495d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:58:26 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:34662
I20260812 00:58:26.037600 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 25423
I20260812 00:58:26.055370 26491 tablet_replica.cc:330] T ad75be9a79b94d21805860412dc4495d P 5848e5a597064322aa7c5b615f34e583: stopping tablet replica
I20260812 00:58:26.055655 26491 raft_consensus.cc:2229] T ad75be9a79b94d21805860412dc4495d P 5848e5a597064322aa7c5b615f34e583 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:58:26.056043 26491 raft_consensus.cc:2258] T ad75be9a79b94d21805860412dc4495d P 5848e5a597064322aa7c5b615f34e583 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:58:26.057421 26489 tablet_replica.cc:330] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244: stopping tablet replica
I20260812 00:58:26.057852 26489 raft_consensus.cc:2229] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:58:26.058350 26491 ts_tablet_manager.cc:1905] T ad75be9a79b94d21805860412dc4495d P 5848e5a597064322aa7c5b615f34e583: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:58:26.058653 26489 raft_consensus.cc:2258] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:58:26.062536 26489 ts_tablet_manager.cc:1905] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 12 00:58:26 dist-test-slave-7f01 krb5kdc[25213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496303, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.201.65@KRBTEST.COM
I20260812 00:58:26.067346 26491 ts_tablet_manager.cc:1918] T ad75be9a79b94d21805860412dc4495d P 5848e5a597064322aa7c5b615f34e583: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 00:58:26.067834 26491 log.cc:1196] T ad75be9a79b94d21805860412dc4495d P 5848e5a597064322aa7c5b615f34e583: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/ad75be9a79b94d21805860412dc4495d
I20260812 00:58:26.068764 26491 ts_tablet_manager.cc:1939] T ad75be9a79b94d21805860412dc4495d P 5848e5a597064322aa7c5b615f34e583: Deleting consensus metadata
I20260812 00:58:26.070649 25262 catalog_manager.cc:4841] TS 5848e5a597064322aa7c5b615f34e583 (127.21.201.67:40315): tablet ad75be9a79b94d21805860412dc4495d (table test-table [id=2f22e94d18704c72abfa7bd46f7a0755]) successfully deleted
I20260812 00:58:26.072530 26489 ts_tablet_manager.cc:1918] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 00:58:26.072573 26490 tablet_replica.cc:330] T ad75be9a79b94d21805860412dc4495d P 0dbc61bb753e46a9be28f8e789ad8b28: stopping tablet replica
I20260812 00:58:26.072731 26489 log.cc:1196] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/ad75be9a79b94d21805860412dc4495d
I20260812 00:58:26.072834 26490 raft_consensus.cc:2229] T ad75be9a79b94d21805860412dc4495d P 0dbc61bb753e46a9be28f8e789ad8b28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:58:26.073136 26490 raft_consensus.cc:2258] T ad75be9a79b94d21805860412dc4495d P 0dbc61bb753e46a9be28f8e789ad8b28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:58:26.073300 26489 ts_tablet_manager.cc:1939] T ad75be9a79b94d21805860412dc4495d P ad6137c90eb04e17bccecb4fa4977244: Deleting consensus metadata
I20260812 00:58:26.074615 26490 ts_tablet_manager.cc:1905] T ad75be9a79b94d21805860412dc4495d P 0dbc61bb753e46a9be28f8e789ad8b28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:58:26.074613 25263 catalog_manager.cc:4841] TS ad6137c90eb04e17bccecb4fa4977244 (127.21.201.66:32843): tablet ad75be9a79b94d21805860412dc4495d (table test-table [id=2f22e94d18704c72abfa7bd46f7a0755]) successfully deleted
I20260812 00:58:26.080331 26493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:26.039810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58792 (local address 127.21.201.65:44581)
0812 00:58:26.052559 (+ 12749us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:26.052570 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:26.052622 (+ 52us) server_negotiation.cc:407] Connection header received
0812 00:58:26.052740 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:26.052750 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:26.052902 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:26.053121 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:26.057653 (+ 4532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:26.059130 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:26.061197 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:26.061896 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:26.062063 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:26.070097 (+ 8034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:26.070132 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:26.070163 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:26.070228 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:26.076078 (+ 5850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:26.077283 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:26.077294 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:26.077298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:26.077593 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:26.078219 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:26.078223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:26.078226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:26.078850 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:26.079198 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:26.079461 (+ 263us) server_negotiation.cc:299] Negotiation successful
0812 00:58:26.079827 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12509,"thread_start_us":12272,"threads_started":1}
W20260812 00:58:26.081297 25654 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58792
W20260812 00:58:26.082366 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 25423: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 00:58:26.082722 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 25423
W20260812 00:58:26.127312 25258 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.201.65:44581 (error 108)
I20260812 00:58:26.127511 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 25672
I20260812 00:58:26.130591 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:26.129302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:44581 (local address 127.0.0.1:58804)
0812 00:58:26.129939 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:26.130193 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":329,"threads_started":1}
W20260812 00:58:26.130936 25258 catalog_manager.cc:4584] TS 0dbc61bb753e46a9be28f8e789ad8b28 (127.21.201.65:44581): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ad75be9a79b94d21805860412dc4495d: Network error: Client connection negotiation failed: client connection to 127.21.201.65:44581: connect: Connection refused (error 111)
Aug 12 00:58:26 dist-test-slave-7f01 krb5kdc[25213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496303, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.201.66@KRBTEST.COM
I20260812 00:58:26.149900 26498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:26.129043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34694 (local address 127.21.201.66:32843)
0812 00:58:26.129758 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:26.129764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:26.129788 (+ 24us) server_negotiation.cc:407] Connection header received
0812 00:58:26.129867 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:26.129873 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:26.129964 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:26.130083 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:26.132860 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:26.133754 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:26.135438 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:26.136025 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:26.136152 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:26.141316 (+ 5164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:26.141337 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:26.141355 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:26.141409 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:26.145488 (+ 4079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:26.146814 (+ 1326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:26.146824 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:26.146826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:26.147112 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:26.147664 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:26.147671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:26.147673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:26.148490 (+ 817us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:26.148793 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:26.149217 (+ 424us) server_negotiation.cc:299] Negotiation successful
0812 00:58:26.149457 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":320,"threads_started":1}
W20260812 00:58:26.150933 25899 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34694
W20260812 00:58:26.152019 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 25672: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 00:58:26.152205 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 25672
I20260812 00:58:26.185416 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:26.184888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:44581 (local address 127.0.0.1:58812)
0812 00:58:26.185110 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:26.185237 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260812 00:58:26.186715 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:26.186251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:44581 (local address 127.0.0.1:58824)
0812 00:58:26.186423 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:26.186550 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260812 00:58:26.204495 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 25912
Aug 12 00:58:26 dist-test-slave-7f01 krb5kdc[25213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496303, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.201.67@KRBTEST.COM
I20260812 00:58:26.226132 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:26.205768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36946 (local address 127.21.201.67:40315)
0812 00:58:26.206588 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:26.206594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:26.206622 (+ 28us) server_negotiation.cc:407] Connection header received
0812 00:58:26.206701 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:26.206706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:26.206811 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:26.206951 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:26.208874 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:26.209840 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:26.211676 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:26.212278 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:26.212408 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:26.217697 (+ 5289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:26.217718 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:26.217735 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:26.217796 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:26.221668 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:26.222656 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:26.222670 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:26.222673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:26.222958 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:26.223703 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:26.223709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:26.223712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:26.224281 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:26.224565 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:26.225241 (+ 676us) server_negotiation.cc:299] Negotiation successful
0812 00:58:26.225616 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":455,"threads_started":1}
W20260812 00:58:26.227222 26162 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36946
W20260812 00:58:26.228199 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 25912: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 00:58:26.228506 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 25912
I20260812 00:58:26.247851 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:26.247263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:44581 (local address 127.0.0.1:58830)
0812 00:58:26.247498 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:26.247602 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260812 00:58:26.248982 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:26.248591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:44581 (local address 127.0.0.1:58834)
0812 00:58:26.248757 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:26.248837 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260812 00:58:26.278035 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 25236
I20260812 00:58:26.279595 25362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:26.410569 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:26.409938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:44581 (local address 127.0.0.1:58844)
0812 00:58:26.410271 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:26.410392 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260812 00:58:26.410627 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 25236
2026-08-12T00:58:26Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (10487 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:58:26 dist-test-slave-7f01 krb5kdc[26510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:58:26 dist-test-slave-7f01 krb5kdc[26510](info): set up 2 sockets
Aug 12 00:58:26 dist-test-slave-7f01 krb5kdc[26510](info): commencing operation
krb5kdc: starting...
W20260812 00:58:27.831729 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.358s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 00:58:28 dist-test-slave-7f01 krb5kdc[26510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:58:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:58:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 00:58:28.113806 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:46325
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:46325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:28.492223 26528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:28.492595 26528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:28.492743 26528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:28.500698 26528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:58:28.500859 26528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:28.500912 26528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:28.501012 26528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:58:28.501099 26528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:58:28.511961 26528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:46325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:46325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:28.513520 26528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:28.515029 26528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:28.529270 26543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:28.529314 26541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:28.529364 26538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:28.530536 26528 server_base.cc:1034] running on GCE node
I20260812 00:58:28.531553 26528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:28.533352 26528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:28.534653 26528 hybrid_clock.cc:648] HybridClock initialized: now 1786496308534606 us; error 56 us; skew 500 ppm
I20260812 00:58:28.535073 26528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:28 dist-test-slave-7f01 krb5kdc[26510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496308, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:28.538956 26528 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 00:58:28.540920 26528 webserver.cc:460] Webserver started at http://127.21.201.126:34671/ using document root <none> and password file <none>
I20260812 00:58:28.541728 26528 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:28.541896 26528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:28.542253 26528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:28.545787 26528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "127881c8fd2c4c4ab476613f1015bc9b"
format_stamp: "Formatted at 2026-08-12 00:58:28 on dist-test-slave-7f01"
server_key: "081d53c3f26d2ade8d6185e36f7923f4"
server_key_iv: "00057a95aa8b2d98cb067297f9969421"
server_key_version: "encryptionkey@0"
I20260812 00:58:28.546726 26528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "127881c8fd2c4c4ab476613f1015bc9b"
format_stamp: "Formatted at 2026-08-12 00:58:28 on dist-test-slave-7f01"
server_key: "081d53c3f26d2ade8d6185e36f7923f4"
server_key_iv: "00057a95aa8b2d98cb067297f9969421"
server_key_version: "encryptionkey@0"
I20260812 00:58:28.553094 26528 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 00:58:28.557190 26564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:28.559648 26528 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 00:58:28.560070 26528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "127881c8fd2c4c4ab476613f1015bc9b"
format_stamp: "Formatted at 2026-08-12 00:58:28 on dist-test-slave-7f01"
server_key: "081d53c3f26d2ade8d6185e36f7923f4"
server_key_iv: "00057a95aa8b2d98cb067297f9969421"
server_key_version: "encryptionkey@0"
I20260812 00:58:28.560398 26528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:28.589082 26528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:28.594214 26528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:28.594568 26528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:28.618480 26528 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:46325
I20260812 00:58:28.618520 26650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:46325 every 8 connection(s)
I20260812 00:58:28.620069 26528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 00:58:28.621464 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 26528
I20260812 00:58:28.621733 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 00:58:28.622603 22309 external_mini_cluster.cc:1428] Setting key 223779e9d84700f4a74bafc9455309de
I20260812 00:58:28.625666 26651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 00:58:28 dist-test-slave-7f01 krb5kdc[26510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:28.642982 26651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 127881c8fd2c4c4ab476613f1015bc9b: Bootstrap starting.
I20260812 00:58:28.647722 26651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 127881c8fd2c4c4ab476613f1015bc9b: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:28.649206 26651 log.cc:824] T 00000000000000000000000000000000 P 127881c8fd2c4c4ab476613f1015bc9b: Log is configured to *not* fsync() on all Append() calls
I20260812 00:58:28.654511 26651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 127881c8fd2c4c4ab476613f1015bc9b: No bootstrap required, opened a new log
I20260812 00:58:28.655979 26660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:28.625568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45266 (local address 127.21.201.126:46325)
0812 00:58:28.626804 (+ 1236us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:28.626820 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:28.626881 (+ 61us) server_negotiation.cc:407] Connection header received
0812 00:58:28.629076 (+ 2195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:28.629107 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:28.629701 (+ 594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:28.630218 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:28.632866 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:28.635184 (+ 2318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:28.637200 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:28.638602 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:28.638867 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:28.643858 (+ 4991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:28.643872 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:28.643904 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:28.643962 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:28.648676 (+ 4714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:28.650141 (+ 1465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:28.650149 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:28.650156 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:28.650398 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:28.651202 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:28.651207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:28.651210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:28.652329 (+ 1119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:28.652755 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:28.653415 (+ 660us) server_negotiation.cc:299] Negotiation successful
0812 00:58:28.654045 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":940,"thread_start_us":577,"threads_started":1}
I20260812 00:58:28.664991 26651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 127881c8fd2c4c4ab476613f1015bc9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127881c8fd2c4c4ab476613f1015bc9b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46325 } }
I20260812 00:58:28.665483 26651 raft_consensus.cc:374] T 00000000000000000000000000000000 P 127881c8fd2c4c4ab476613f1015bc9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:28.665637 26651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 127881c8fd2c4c4ab476613f1015bc9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 127881c8fd2c4c4ab476613f1015bc9b, State: Initialized, Role: FOLLOWER
I20260812 00:58:28.666353 26651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 127881c8fd2c4c4ab476613f1015bc9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127881c8fd2c4c4ab476613f1015bc9b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46325 } }
I20260812 00:58:28.666604 26651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 127881c8fd2c4c4ab476613f1015bc9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:58:28.666811 26651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 127881c8fd2c4c4ab476613f1015bc9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:58:28.666981 26651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 127881c8fd2c4c4ab476613f1015bc9b [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:28.669590 26651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 127881c8fd2c4c4ab476613f1015bc9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127881c8fd2c4c4ab476613f1015bc9b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46325 } }
I20260812 00:58:28.670295 26651 leader_election.cc:302] T 00000000000000000000000000000000 P 127881c8fd2c4c4ab476613f1015bc9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 127881c8fd2c4c4ab476613f1015bc9b; no voters:
I20260812 00:58:28.671149 26651 leader_election.cc:288] T 00000000000000000000000000000000 P 127881c8fd2c4c4ab476613f1015bc9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:58:28.671291 26662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 127881c8fd2c4c4ab476613f1015bc9b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:58:28.672042 26662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 127881c8fd2c4c4ab476613f1015bc9b [term 1 LEADER]: Becoming Leader. State: Replica: 127881c8fd2c4c4ab476613f1015bc9b, State: Running, Role: LEADER
I20260812 00:58:28.672719 26662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 127881c8fd2c4c4ab476613f1015bc9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127881c8fd2c4c4ab476613f1015bc9b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46325 } }
I20260812 00:58:28.673848 26651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 127881c8fd2c4c4ab476613f1015bc9b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:58:28.676221 26665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 127881c8fd2c4c4ab476613f1015bc9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 127881c8fd2c4c4ab476613f1015bc9b. Latest consensus state: current_term: 1 leader_uuid: "127881c8fd2c4c4ab476613f1015bc9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127881c8fd2c4c4ab476613f1015bc9b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46325 } } }
I20260812 00:58:28.676683 26665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 127881c8fd2c4c4ab476613f1015bc9b [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:28.679852 26664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 127881c8fd2c4c4ab476613f1015bc9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "127881c8fd2c4c4ab476613f1015bc9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127881c8fd2c4c4ab476613f1015bc9b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46325 } } }
I20260812 00:58:28.680294 26664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 127881c8fd2c4c4ab476613f1015bc9b [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:28.681106 26673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:58:28.686937 26673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:58:28.698280 26673 catalog_manager.cc:1347] Generated new cluster ID: 1b4244352850433ab5f0df3bd223b410
I20260812 00:58:28.698449 26673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:58:28.735826 26673 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:58:28.737406 26673 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:58:28.748350 26673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 127881c8fd2c4c4ab476613f1015bc9b: Generated new TSK 0
I20260812 00:58:28.750121 26673 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.201.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
I20260812 00:58:28.829020 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:0
--local_ip_for_outbound_sockets=127.21.201.65
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:46325
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:29.241737 26704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:29.242179 26704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:29.242288 26704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:29.250463 26704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:29.250648 26704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:29.250774 26704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 00:58:29.263052 26704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:46325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:29.264750 26704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:29.266399 26704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:29.282385 26717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:29.282373 26720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:29.282763 26704 server_base.cc:1034] running on GCE node
W20260812 00:58:29.283223 26713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:29.284219 26704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:29.285498 26704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:29.286870 26704 hybrid_clock.cc:648] HybridClock initialized: now 1786496309286773 us; error 83 us; skew 500 ppm
I20260812 00:58:29.287324 26704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:29 dist-test-slave-7f01 krb5kdc[26510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496309, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:29.291913 26704 init.cc:377] Logged in from keytab as kudu/127.21.201.65@KRBTEST.COM (short username kudu)
I20260812 00:58:29.293992 26704 webserver.cc:460] Webserver started at http://127.21.201.65:33495/ using document root <none> and password file <none>
I20260812 00:58:29.294898 26704 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:29.295020 26704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:29.295572 26704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:29.299335 26704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/data/instance:
uuid: "68d7750a2d014b1bb22a08f8ff62f01b"
format_stamp: "Formatted at 2026-08-12 00:58:29 on dist-test-slave-7f01"
server_key: "cef0ece0f1d4999281f9463c4aa52d9b"
server_key_iv: "0e1ded1a093ddea4be5b079c29a93cd2"
server_key_version: "encryptionkey@0"
I20260812 00:58:29.300504 26704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance:
uuid: "68d7750a2d014b1bb22a08f8ff62f01b"
format_stamp: "Formatted at 2026-08-12 00:58:29 on dist-test-slave-7f01"
server_key: "cef0ece0f1d4999281f9463c4aa52d9b"
server_key_iv: "0e1ded1a093ddea4be5b079c29a93cd2"
server_key_version: "encryptionkey@0"
I20260812 00:58:29.306941 26704 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 00:58:29.312011 26735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:29.314174 26704 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260812 00:58:29.314392 26704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "68d7750a2d014b1bb22a08f8ff62f01b"
format_stamp: "Formatted at 2026-08-12 00:58:29 on dist-test-slave-7f01"
server_key: "cef0ece0f1d4999281f9463c4aa52d9b"
server_key_iv: "0e1ded1a093ddea4be5b079c29a93cd2"
server_key_version: "encryptionkey@0"
I20260812 00:58:29.314613 26704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:29.342028 26704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:29.348472 26704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:29.348837 26704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:29.350267 26704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:58:29.352620 26704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:29.352753 26704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:29.352864 26704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:29.352922 26704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:29.394433 26704 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:43545
I20260812 00:58:29.394490 26908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:43545 every 8 connection(s)
I20260812 00:58:29.395926 26704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 00:58:29.401587 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 26704
I20260812 00:58:29.401901 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance
I20260812 00:58:29.402714 22309 external_mini_cluster.cc:1428] Setting key e4dac6cadbfeb3b8abd36c16608f07b1
Aug 12 00:58:29 dist-test-slave-7f01 krb5kdc[26510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496309, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:29.419848 26914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:29.398801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:53053 (local address 127.21.201.126:46325)
0812 00:58:29.399333 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:29.399338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:29.400451 (+ 1113us) server_negotiation.cc:407] Connection header received
0812 00:58:29.402623 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:29.402630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:29.402738 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:29.402871 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:29.404873 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:29.405864 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:29.407659 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:29.408196 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:29.408343 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:29.412684 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:29.412690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:29.412693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:29.412733 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:29.416148 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:29.417110 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:29.417115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:29.417117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:29.417270 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:29.417767 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:29.417771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:29.417773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:29.418073 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:29.418245 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:29.419120 (+ 875us) server_negotiation.cc:299] Negotiation successful
0812 00:58:29.419431 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":297,"threads_started":1}
I20260812 00:58:29.421010 26915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:29.399409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46325 (local address 127.21.201.65:53053)
0812 00:58:29.400251 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:29.400292 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:29.402303 (+ 2011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:29.403152 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:29.403161 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:29.404002 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:29.404670 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:29.404690 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:29.406078 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:29.406083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:29.407484 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:29.407499 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:29.408474 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:29.408482 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:29.408767 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:29.409654 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:29.409680 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:29.412380 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:29.416358 (+ 3978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:29.416370 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:29.416398 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:29.416947 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:29.417398 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:29.417403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:29.417407 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:29.417628 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:29.418223 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:29.418232 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:29.418635 (+ 403us) client_negotiation.cc:769] Sending connection context
0812 00:58:29.419008 (+ 373us) client_negotiation.cc:240] Negotiation successful
0812 00:58:29.419641 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":364,"threads_started":1}
I20260812 00:58:29.423023 26911 heartbeater.cc:347] Connected to a master server at 127.21.201.126:46325
I20260812 00:58:29.423431 26911 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:29.424544 26911 heartbeater.cc:511] Master 127.21.201.126:46325 requested a full tablet report, sending...
I20260812 00:58:29.427702 26594 ts_manager.cc:194] Registered new tserver with Master: 68d7750a2d014b1bb22a08f8ff62f01b (127.21.201.65:43545)
WARNING: no policy specified for kudu/127.21.201.66@KRBTEST.COM; defaulting to no policy
I20260812 00:58:29.430039 26594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.65@KRBTEST.COM'} at 127.21.201.65:53053
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
I20260812 00:58:29.469522 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:0
--local_ip_for_outbound_sockets=127.21.201.66
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:46325
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:29.851325 26918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:29.851693 26918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:29.851881 26918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:29.859854 26918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:29.860018 26918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:29.860224 26918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 00:58:29.872036 26918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:46325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:29.874167 26918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:29.876432 26918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:29.893759 26925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:29.894495 26927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:29.896247 26918 server_base.cc:1034] running on GCE node
W20260812 00:58:29.896544 26932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:29.897330 26918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:29.898581 26918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:29.899883 26918 hybrid_clock.cc:648] HybridClock initialized: now 1786496309899829 us; error 45 us; skew 500 ppm
I20260812 00:58:29.900375 26918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:29 dist-test-slave-7f01 krb5kdc[26510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496309, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:29.904167 26918 init.cc:377] Logged in from keytab as kudu/127.21.201.66@KRBTEST.COM (short username kudu)
I20260812 00:58:29.906291 26918 webserver.cc:460] Webserver started at http://127.21.201.66:32845/ using document root <none> and password file <none>
I20260812 00:58:29.907208 26918 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:29.907348 26918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:29.907711 26918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:29.911123 26918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/data/instance:
uuid: "0a8b8a714f1e4a2698e55ac148f25d77"
format_stamp: "Formatted at 2026-08-12 00:58:29 on dist-test-slave-7f01"
server_key: "8815647c2dd1da158f69dd30f15baf79"
server_key_iv: "1fe051dcf502dc1453a9e4a11494dbd9"
server_key_version: "encryptionkey@0"
I20260812 00:58:29.912104 26918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance:
uuid: "0a8b8a714f1e4a2698e55ac148f25d77"
format_stamp: "Formatted at 2026-08-12 00:58:29 on dist-test-slave-7f01"
server_key: "8815647c2dd1da158f69dd30f15baf79"
server_key_iv: "1fe051dcf502dc1453a9e4a11494dbd9"
server_key_version: "encryptionkey@0"
I20260812 00:58:29.918381 26918 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260812 00:58:29.923272 26943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:29.925621 26918 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 00:58:29.925943 26918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "0a8b8a714f1e4a2698e55ac148f25d77"
format_stamp: "Formatted at 2026-08-12 00:58:29 on dist-test-slave-7f01"
server_key: "8815647c2dd1da158f69dd30f15baf79"
server_key_iv: "1fe051dcf502dc1453a9e4a11494dbd9"
server_key_version: "encryptionkey@0"
I20260812 00:58:29.926273 26918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:29.947974 26918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:29.953073 26918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:29.953435 26918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:29.954638 26918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:58:29.956347 26918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:29.956465 26918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:29.956622 26918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:29.956700 26918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:30.000247 26918 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:41333
I20260812 00:58:30.000288 27129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:41333 every 8 connection(s)
I20260812 00:58:30.001724 26918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 00:58:30.009366 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 26918
I20260812 00:58:30.009572 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance
I20260812 00:58:30.010108 22309 external_mini_cluster.cc:1428] Setting key a23f4e5607fbf03fa543f71adb718553
Aug 12 00:58:30 dist-test-slave-7f01 krb5kdc[26510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496309, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:30.030977 27133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:30.005959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:49853 (local address 127.21.201.126:46325)
0812 00:58:30.006642 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:30.006649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:30.008142 (+ 1493us) server_negotiation.cc:407] Connection header received
0812 00:58:30.012125 (+ 3983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:30.012131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:30.012206 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:30.012329 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:30.014419 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:30.015188 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:30.017074 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:30.017547 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:30.017709 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:30.023268 (+ 5559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:30.023274 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:30.023277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:30.023321 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:30.026945 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:30.028100 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:30.028105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:30.028108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:30.028272 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:30.028812 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:30.028822 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:30.028825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:30.029177 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:30.029369 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:30.030207 (+ 838us) server_negotiation.cc:299] Negotiation successful
0812 00:58:30.030507 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":352,"threads_started":1}
I20260812 00:58:30.032330 27134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:30.006636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46325 (local address 127.21.201.66:49853)
0812 00:58:30.007906 (+ 1270us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:30.007962 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:30.011473 (+ 3511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:30.012739 (+ 1266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:30.012754 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:30.013505 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:30.014195 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:30.014216 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:30.015393 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:30.015397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:30.016897 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:30.016913 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:30.017829 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:30.017838 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:30.018117 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:30.018983 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:30.019018 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:30.022912 (+ 3894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:30.027199 (+ 4287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:30.027217 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:30.027234 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:30.027923 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:30.028417 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:30.028423 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:30.028428 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:30.028668 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:30.029330 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:30.029340 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:30.029775 (+ 435us) client_negotiation.cc:769] Sending connection context
0812 00:58:30.030095 (+ 320us) client_negotiation.cc:240] Negotiation successful
0812 00:58:30.030792 (+ 697us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":662,"threads_started":1}
WARNING: no policy specified for kudu/127.21.201.67@KRBTEST.COM; defaulting to no policy
I20260812 00:58:30.034217 27130 heartbeater.cc:347] Connected to a master server at 127.21.201.126:46325
I20260812 00:58:30.034601 27130 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:30.035661 27130 heartbeater.cc:511] Master 127.21.201.126:46325 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.67@KRBTEST.COM" created.
I20260812 00:58:30.038084 26594 ts_manager.cc:194] Registered new tserver with Master: 0a8b8a714f1e4a2698e55ac148f25d77 (127.21.201.66:41333)
I20260812 00:58:30.039577 26594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.66@KRBTEST.COM'} at 127.21.201.66:49853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
I20260812 00:58:30.072120 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:0
--local_ip_for_outbound_sockets=127.21.201.67
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:46325
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:58:30.434814 26911 heartbeater.cc:503] Master 127.21.201.126:46325 was elected leader, sending a full tablet report...
W20260812 00:58:30.481416 27139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:30.481776 27139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:30.481961 27139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:30.490168 27139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:30.490307 27139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:30.490486 27139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 00:58:30.501339 27139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:46325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:30.502954 27139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:30.505158 27139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:30.520936 27151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:30.521064 27148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:30.521064 27155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:30.523015 27139 server_base.cc:1034] running on GCE node
I20260812 00:58:30.524220 27139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:30.525740 27139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:30.527071 27139 hybrid_clock.cc:648] HybridClock initialized: now 1786496310526989 us; error 65 us; skew 500 ppm
I20260812 00:58:30.527673 27139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:30 dist-test-slave-7f01 krb5kdc[26510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496310, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:30.532145 27139 init.cc:377] Logged in from keytab as kudu/127.21.201.67@KRBTEST.COM (short username kudu)
I20260812 00:58:30.534565 27139 webserver.cc:460] Webserver started at http://127.21.201.67:40851/ using document root <none> and password file <none>
I20260812 00:58:30.535401 27139 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:30.535532 27139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:30.535923 27139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:30.539529 27139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/data/instance:
uuid: "dc1dcd528cfd432d934e1f8974ee5c65"
format_stamp: "Formatted at 2026-08-12 00:58:30 on dist-test-slave-7f01"
server_key: "40876f15aac76c14358f4233b2220ff6"
server_key_iv: "2ab8ccd948b904e04f571c28da9399d2"
server_key_version: "encryptionkey@0"
I20260812 00:58:30.540571 27139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance:
uuid: "dc1dcd528cfd432d934e1f8974ee5c65"
format_stamp: "Formatted at 2026-08-12 00:58:30 on dist-test-slave-7f01"
server_key: "40876f15aac76c14358f4233b2220ff6"
server_key_iv: "2ab8ccd948b904e04f571c28da9399d2"
server_key_version: "encryptionkey@0"
I20260812 00:58:30.547503 27139 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 00:58:30.552315 27176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:30.554172 27139 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 00:58:30.554414 27139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "dc1dcd528cfd432d934e1f8974ee5c65"
format_stamp: "Formatted at 2026-08-12 00:58:30 on dist-test-slave-7f01"
server_key: "40876f15aac76c14358f4233b2220ff6"
server_key_iv: "2ab8ccd948b904e04f571c28da9399d2"
server_key_version: "encryptionkey@0"
I20260812 00:58:30.554708 27139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:30.574642 27139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:30.579952 27139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:30.580310 27139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:30.581557 27139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:58:30.583329 27139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:30.583418 27139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:30.583580 27139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:30.583636 27139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:30.625830 27139 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:43143
I20260812 00:58:30.625901 27361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:43143 every 8 connection(s)
I20260812 00:58:30.627697 27139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 00:58:30.633018 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 27139
I20260812 00:58:30.633262 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance
I20260812 00:58:30.633869 22309 external_mini_cluster.cc:1428] Setting key 6aad453f80ed463e1fa56819980825dc
Aug 12 00:58:30 dist-test-slave-7f01 krb5kdc[26510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496310, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:30.655550 27367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:30.631066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:36617 (local address 127.21.201.126:46325)
0812 00:58:30.631597 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:30.631602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:30.633014 (+ 1412us) server_negotiation.cc:407] Connection header received
0812 00:58:30.635672 (+ 2658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:30.635681 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:30.635800 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:30.635901 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:30.638023 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:30.638954 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:30.640939 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:30.641414 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:30.641543 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:30.647051 (+ 5508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:30.647058 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:30.647062 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:30.647114 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:30.651199 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:30.652493 (+ 1294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:30.652499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:30.652501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:30.652677 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:30.653504 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:30.653509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:30.653511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:30.653837 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:30.654016 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:30.654971 (+ 955us) server_negotiation.cc:299] Negotiation successful
0812 00:58:30.655206 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":267,"threads_started":1}
I20260812 00:58:30.656569 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:30.631570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46325 (local address 127.21.201.67:36617)
0812 00:58:30.632778 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:30.632845 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:30.635271 (+ 2426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:30.636251 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:30.636262 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:30.636975 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:30.637818 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:30.637850 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:30.639322 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:30.639326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:30.640767 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:30.640781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:30.641680 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:30.641686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:30.641903 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:30.643156 (+ 1253us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:30.643193 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:30.646641 (+ 3448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:30.651452 (+ 4811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:30.651465 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:30.651488 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:30.652284 (+ 796us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:30.652879 (+ 595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:30.652885 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:30.652888 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:30.653322 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:30.654086 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:30.654094 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:30.654542 (+ 448us) client_negotiation.cc:769] Sending connection context
0812 00:58:30.654826 (+ 284us) client_negotiation.cc:240] Negotiation successful
0812 00:58:30.655300 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"thread_start_us":596,"threads_started":1}
I20260812 00:58:30.658308 27363 heartbeater.cc:347] Connected to a master server at 127.21.201.126:46325
I20260812 00:58:30.658723 27363 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:30.660177 27363 heartbeater.cc:511] Master 127.21.201.126:46325 requested a full tablet report, sending...
I20260812 00:58:30.662585 26594 ts_manager.cc:194] Registered new tserver with Master: dc1dcd528cfd432d934e1f8974ee5c65 (127.21.201.67:43143)
I20260812 00:58:30.664065 26594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.67@KRBTEST.COM'} at 127.21.201.67:36617
I20260812 00:58:30.665298 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 00:58:30.708495 27367 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:58:30.705407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45276 (local address 127.21.201.126:46325)
0812 00:58:30.705680 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:30.705690 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:30.706160 (+ 470us) server_negotiation.cc:407] Connection header received
0812 00:58:30.706395 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:30.706401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:30.706513 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:30.706638 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:30.708258 (+ 1620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45276: BlockingRecv error: recv got EOF from 127.0.0.1:45276 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 00:58:30.713721 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 26704
W20260812 00:58:30.717131 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 26704: Not authorized: Client connection negotiation failed: client connection to 127.21.201.65:43545: server requires authentication, but client does not have Kerberos credentials available
I20260812 00:58:30.717339 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 26704
I20260812 00:58:30.754452 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 26918
W20260812 00:58:30.757632 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 26918: Not authorized: Client connection negotiation failed: client connection to 127.21.201.66:41333: server requires authentication, but client does not have Kerberos credentials available
I20260812 00:58:30.757863 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 26918
I20260812 00:58:30.787201 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 27139
W20260812 00:58:30.791674 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 27139: Not authorized: Client connection negotiation failed: client connection to 127.21.201.67:43143: server requires authentication, but client does not have Kerberos credentials available
I20260812 00:58:30.792017 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 27139
I20260812 00:58:30.818524 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 26528
I20260812 00:58:30.820065 26643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:30.940505 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 26528
2026-08-12T00:58:30Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4531 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:58:31 dist-test-slave-7f01 krb5kdc[27397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:58:31 dist-test-slave-7f01 krb5kdc[27397](info): set up 2 sockets
Aug 12 00:58:31 dist-test-slave-7f01 krb5kdc[27397](info): commencing operation
krb5kdc: starting...
W20260812 00:58:32.341656 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.336s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 00:58:32 dist-test-slave-7f01 krb5kdc[27397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:58:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:58:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 00:58:32.624833 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:45445
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:43923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:45445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:33.025858 27416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:33.026288 27416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:33.026516 27416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:33.034390 27416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:58:33.034601 27416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:33.034720 27416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:33.034832 27416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:58:33.034950 27416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:58:33.046208 27416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:43923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:45445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:45445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:33.047886 27416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:33.049461 27416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:33.065779 27421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:33.065665 27422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:33.065979 27424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:33.067281 27416 server_base.cc:1034] running on GCE node
I20260812 00:58:33.068441 27416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:33.070312 27416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:33.071597 27416 hybrid_clock.cc:648] HybridClock initialized: now 1786496313071553 us; error 52 us; skew 500 ppm
I20260812 00:58:33.072175 27416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:33 dist-test-slave-7f01 krb5kdc[27397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496313, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:33.076117 27416 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 00:58:33.078389 27416 webserver.cc:460] Webserver started at http://127.21.201.126:44325/ using document root <none> and password file <none>
I20260812 00:58:33.079432 27416 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:33.079888 27416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:33.080423 27416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:33.084717 27416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "37cc4ef926464fb9a3e95e69066c617f"
format_stamp: "Formatted at 2026-08-12 00:58:33 on dist-test-slave-7f01"
server_key: "e2e52a2c7fa96668da4ebb9ffa6f2086"
server_key_iv: "1ed6c0fdd191f69155fa4920aa7a2c56"
server_key_version: "encryptionkey@0"
I20260812 00:58:33.085767 27416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "37cc4ef926464fb9a3e95e69066c617f"
format_stamp: "Formatted at 2026-08-12 00:58:33 on dist-test-slave-7f01"
server_key: "e2e52a2c7fa96668da4ebb9ffa6f2086"
server_key_iv: "1ed6c0fdd191f69155fa4920aa7a2c56"
server_key_version: "encryptionkey@0"
I20260812 00:58:33.093231 27416 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20260812 00:58:33.098737 27442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:33.100807 27416 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 00:58:33.101078 27416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "37cc4ef926464fb9a3e95e69066c617f"
format_stamp: "Formatted at 2026-08-12 00:58:33 on dist-test-slave-7f01"
server_key: "e2e52a2c7fa96668da4ebb9ffa6f2086"
server_key_iv: "1ed6c0fdd191f69155fa4920aa7a2c56"
server_key_version: "encryptionkey@0"
I20260812 00:58:33.101328 27416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:33.129766 27416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:33.135545 27416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:33.135936 27416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:33.161512 27416 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:45445
I20260812 00:58:33.161546 27516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:45445 every 8 connection(s)
I20260812 00:58:33.163209 27416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 00:58:33.167539 27517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:33.168704 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 27416
I20260812 00:58:33.168973 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 00:58:33.169857 22309 external_mini_cluster.cc:1428] Setting key c8cf000655834c42f06491b5d0450aac
I20260812 00:58:33.182649 27517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f: Bootstrap starting.
I20260812 00:58:33.187338 27517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:33.188911 27517 log.cc:824] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f: Log is configured to *not* fsync() on all Append() calls
Aug 12 00:58:33 dist-test-slave-7f01 krb5kdc[27397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:33.193528 27517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f: No bootstrap required, opened a new log
I20260812 00:58:33.203411 27523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:33.172954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48488 (local address 127.21.201.126:45445)
0812 00:58:33.174196 (+ 1242us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:33.174213 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:33.174283 (+ 70us) server_negotiation.cc:407] Connection header received
0812 00:58:33.176604 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:33.176641 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:33.177261 (+ 620us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:33.177754 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:33.180082 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:33.181863 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:33.183953 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:33.184813 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:33.185000 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:33.192009 (+ 7009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:33.192021 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:33.192047 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:33.192104 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:33.197146 (+ 5042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:33.198737 (+ 1591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:33.198748 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:33.198763 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:33.199081 (+ 318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:33.199899 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:33.199905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:33.199907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:33.200729 (+ 822us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:33.201135 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:33.201626 (+ 491us) server_negotiation.cc:299] Negotiation successful
0812 00:58:33.202175 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":920,"thread_start_us":568,"threads_started":1}
I20260812 00:58:33.204097 27517 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37cc4ef926464fb9a3e95e69066c617f" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 45445 } }
I20260812 00:58:33.205000 27517 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:33.205170 27517 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37cc4ef926464fb9a3e95e69066c617f, State: Initialized, Role: FOLLOWER
I20260812 00:58:33.206480 27517 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37cc4ef926464fb9a3e95e69066c617f" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 45445 } }
I20260812 00:58:33.207034 27528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:58:33.207427 27528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37cc4ef926464fb9a3e95e69066c617f" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 45445 } }
I20260812 00:58:33.208178 27528 leader_election.cc:302] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37cc4ef926464fb9a3e95e69066c617f; no voters:
I20260812 00:58:33.206976 27517 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:58:33.209164 27528 leader_election.cc:288] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260812 00:58:33.209215 27517 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:58:33.209426 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:58:33.209424 27517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:33.213063 27517 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37cc4ef926464fb9a3e95e69066c617f" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 45445 } }
I20260812 00:58:33.213428 27517 leader_election.cc:302] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37cc4ef926464fb9a3e95e69066c617f; no voters:
I20260812 00:58:33.213538 27529 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260812 00:58:33.213717 27517 leader_election.cc:288] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:58:33.214919 27529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37cc4ef926464fb9a3e95e69066c617f" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 45445 } } }
I20260812 00:58:33.215468 27529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:58:33.215729 27528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:58:33.216066 27528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f [term 1 LEADER]: Becoming Leader. State: Replica: 37cc4ef926464fb9a3e95e69066c617f, State: Running, Role: LEADER
I20260812 00:58:33.216758 27528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37cc4ef926464fb9a3e95e69066c617f" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 45445 } }
I20260812 00:58:33.218204 27517 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:58:33.219419 27530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37cc4ef926464fb9a3e95e69066c617f. Latest consensus state: current_term: 1 leader_uuid: "37cc4ef926464fb9a3e95e69066c617f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37cc4ef926464fb9a3e95e69066c617f" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 45445 } } }
I20260812 00:58:33.219899 27530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:33.221596 27535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:58:33.230268 27535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:58:33.263389 27535 catalog_manager.cc:1347] Generated new cluster ID: d47e39c8c27d404dbdf3ee67bed663fb
I20260812 00:58:33.263595 27535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:58:33.302335 27535 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:58:33.304778 27535 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:58:33.312979 27523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:33.290027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48492 (local address 127.21.201.126:45445)
0812 00:58:33.290328 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:33.290338 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:33.300428 (+ 10090us) server_negotiation.cc:407] Connection header received
0812 00:58:33.301138 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:33.301148 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:33.301276 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:33.301376 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:33.303469 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:33.305112 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:33.308559 (+ 3447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:33.309128 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:33.309211 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:33.310624 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:33.310629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:33.310633 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:58:33.311489 (+ 856us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:33.311607 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:58:33.311639 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:33.311791 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:33.312171 (+ 380us) server_negotiation.cc:299] Negotiation successful
0812 00:58:33.312397 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":103}
I20260812 00:58:33.314309 27535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37cc4ef926464fb9a3e95e69066c617f: Generated new TSK 0
I20260812 00:58:33.315371 27535 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:58:33.326933 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 27416
I20260812 00:58:33.328936 27510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:33.428380 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 27416
2026-08-12T00:58:33Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2471 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:58:33 dist-test-slave-7f01 krb5kdc[27574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:58:33 dist-test-slave-7f01 krb5kdc[27574](info): set up 2 sockets
Aug 12 00:58:33 dist-test-slave-7f01 krb5kdc[27574](info): commencing operation
krb5kdc: starting...
W20260812 00:58:34.833479 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.356s user 0.004s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 00:58:35 dist-test-slave-7f01 krb5kdc[27574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:58:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:58:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 00:58:35.120868 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:38153
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:36841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:38153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:35.517267 27592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:35.517784 27592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:35.518002 27592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:35.527302 27592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:58:35.527503 27592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:35.527556 27592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:35.527611 27592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:58:35.527778 27592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:58:35.540025 27592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:36841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:38153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:38153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:35.541615 27592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:35.543999 27592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:35.557901 27598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:35.558444 27602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:35.558513 27599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:35.559243 27592 server_base.cc:1034] running on GCE node
I20260812 00:58:35.560708 27592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:35.562506 27592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:35.563952 27592 hybrid_clock.cc:648] HybridClock initialized: now 1786496315563852 us; error 45 us; skew 500 ppm
I20260812 00:58:35.564603 27592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:35 dist-test-slave-7f01 krb5kdc[27574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496315, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:35.574496 27592 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 00:58:35.576757 27592 webserver.cc:460] Webserver started at http://127.21.201.126:33661/ using document root <none> and password file <none>
I20260812 00:58:35.577945 27592 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:35.578147 27592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:35.578534 27592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:35.582129 27592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "a036b5fc142f4df7858477d82519db8a"
format_stamp: "Formatted at 2026-08-12 00:58:35 on dist-test-slave-7f01"
server_key: "8db1b9a3871aa5f37ecdbddb10dec7f5"
server_key_iv: "7ebd6b5c16a3734da1ca2213d9913c50"
server_key_version: "encryptionkey@0"
I20260812 00:58:35.583019 27592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "a036b5fc142f4df7858477d82519db8a"
format_stamp: "Formatted at 2026-08-12 00:58:35 on dist-test-slave-7f01"
server_key: "8db1b9a3871aa5f37ecdbddb10dec7f5"
server_key_iv: "7ebd6b5c16a3734da1ca2213d9913c50"
server_key_version: "encryptionkey@0"
I20260812 00:58:35.589506 27592 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 00:58:35.594071 27614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:35.595707 27592 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 00:58:35.596032 27592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "a036b5fc142f4df7858477d82519db8a"
format_stamp: "Formatted at 2026-08-12 00:58:35 on dist-test-slave-7f01"
server_key: "8db1b9a3871aa5f37ecdbddb10dec7f5"
server_key_iv: "7ebd6b5c16a3734da1ca2213d9913c50"
server_key_version: "encryptionkey@0"
I20260812 00:58:35.596287 27592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:35.635967 27592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:35.641062 27592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:35.641425 27592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:35.665323 27592 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:38153
I20260812 00:58:35.665446 27697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:38153 every 8 connection(s)
I20260812 00:58:35.667003 27592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 00:58:35.671851 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 27592
I20260812 00:58:35.672102 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 00:58:35.672403 27699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:35.672859 22309 external_mini_cluster.cc:1428] Setting key a79b9389ad308fd954e797f13af4eddf
I20260812 00:58:35.689725 27699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a036b5fc142f4df7858477d82519db8a: Bootstrap starting.
Aug 12 00:58:35 dist-test-slave-7f01 krb5kdc[27574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:35.693434 27699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a036b5fc142f4df7858477d82519db8a: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:35.695083 27699 log.cc:824] T 00000000000000000000000000000000 P a036b5fc142f4df7858477d82519db8a: Log is configured to *not* fsync() on all Append() calls
I20260812 00:58:35.699093 27699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a036b5fc142f4df7858477d82519db8a: No bootstrap required, opened a new log
I20260812 00:58:35.701992 27702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:35.675434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58308 (local address 127.21.201.126:38153)
0812 00:58:35.676404 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:35.676420 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:35.676471 (+ 51us) server_negotiation.cc:407] Connection header received
0812 00:58:35.678306 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:35.678332 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:35.678797 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:35.679184 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:35.681386 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:35.682759 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:35.684662 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:35.685506 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:35.685686 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:35.691593 (+ 5907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:35.691606 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:35.691629 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:35.691695 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:35.696641 (+ 4946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:35.697598 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:35.697606 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:35.697614 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:35.697805 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:35.698530 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:35.698535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:35.698537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:35.699329 (+ 792us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:35.699778 (+ 449us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:35.700190 (+ 412us) server_negotiation.cc:299] Negotiation successful
0812 00:58:35.700706 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":443,"threads_started":1}
I20260812 00:58:35.709378 27699 raft_consensus.cc:348] T 00000000000000000000000000000000 P a036b5fc142f4df7858477d82519db8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a036b5fc142f4df7858477d82519db8a" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38153 } }
I20260812 00:58:35.709821 27699 raft_consensus.cc:374] T 00000000000000000000000000000000 P a036b5fc142f4df7858477d82519db8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:35.709939 27699 raft_consensus.cc:729] T 00000000000000000000000000000000 P a036b5fc142f4df7858477d82519db8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a036b5fc142f4df7858477d82519db8a, State: Initialized, Role: FOLLOWER
I20260812 00:58:35.710644 27699 consensus_queue.cc:260] T 00000000000000000000000000000000 P a036b5fc142f4df7858477d82519db8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a036b5fc142f4df7858477d82519db8a" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38153 } }
I20260812 00:58:35.710896 27699 raft_consensus.cc:388] T 00000000000000000000000000000000 P a036b5fc142f4df7858477d82519db8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:58:35.711074 27699 raft_consensus.cc:482] T 00000000000000000000000000000000 P a036b5fc142f4df7858477d82519db8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:58:35.711294 27699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a036b5fc142f4df7858477d82519db8a [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:35.714275 27699 raft_consensus.cc:504] T 00000000000000000000000000000000 P a036b5fc142f4df7858477d82519db8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a036b5fc142f4df7858477d82519db8a" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38153 } }
I20260812 00:58:35.714890 27699 leader_election.cc:302] T 00000000000000000000000000000000 P a036b5fc142f4df7858477d82519db8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a036b5fc142f4df7858477d82519db8a; no voters:
I20260812 00:58:35.715606 27699 leader_election.cc:288] T 00000000000000000000000000000000 P a036b5fc142f4df7858477d82519db8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:58:35.715842 27704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a036b5fc142f4df7858477d82519db8a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:58:35.716578 27704 raft_consensus.cc:686] T 00000000000000000000000000000000 P a036b5fc142f4df7858477d82519db8a [term 1 LEADER]: Becoming Leader. State: Replica: a036b5fc142f4df7858477d82519db8a, State: Running, Role: LEADER
I20260812 00:58:35.717307 27704 consensus_queue.cc:237] T 00000000000000000000000000000000 P a036b5fc142f4df7858477d82519db8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a036b5fc142f4df7858477d82519db8a" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38153 } }
I20260812 00:58:35.717978 27699 sys_catalog.cc:564] T 00000000000000000000000000000000 P a036b5fc142f4df7858477d82519db8a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:58:35.720808 27706 sys_catalog.cc:455] T 00000000000000000000000000000000 P a036b5fc142f4df7858477d82519db8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a036b5fc142f4df7858477d82519db8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a036b5fc142f4df7858477d82519db8a" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38153 } } }
I20260812 00:58:35.721230 27706 sys_catalog.cc:458] T 00000000000000000000000000000000 P a036b5fc142f4df7858477d82519db8a [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:35.722214 27707 sys_catalog.cc:455] T 00000000000000000000000000000000 P a036b5fc142f4df7858477d82519db8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a036b5fc142f4df7858477d82519db8a. Latest consensus state: current_term: 1 leader_uuid: "a036b5fc142f4df7858477d82519db8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a036b5fc142f4df7858477d82519db8a" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38153 } } }
I20260812 00:58:35.722747 27707 sys_catalog.cc:458] T 00000000000000000000000000000000 P a036b5fc142f4df7858477d82519db8a [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:35.724014 27713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:58:35.729545 27713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:58:35.741303 27713 catalog_manager.cc:1347] Generated new cluster ID: c65896b2427f4af7a4e2f4bdb6e7ce09
I20260812 00:58:35.741484 27713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:58:35.764158 27713 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:58:35.765787 27713 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:58:35.772310 27713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a036b5fc142f4df7858477d82519db8a: Generated new TSK 0
I20260812 00:58:35.773469 27713 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.201.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
I20260812 00:58:35.841821 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:0
--local_ip_for_outbound_sockets=127.21.201.65
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38153
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:36841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:36.265331 27736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:36.265712 27736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:36.265827 27736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:36.273955 27736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:36.274120 27736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:36.274261 27736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 00:58:36.287331 27736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:36841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:36.289301 27736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:36.291015 27736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:36.307156 27750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:36.307291 27755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:36.307292 27752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:36.308595 27736 server_base.cc:1034] running on GCE node
I20260812 00:58:36.309360 27736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:36.310580 27736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:36.311862 27736 hybrid_clock.cc:648] HybridClock initialized: now 1786496316311815 us; error 32 us; skew 500 ppm
I20260812 00:58:36.312520 27736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:36 dist-test-slave-7f01 krb5kdc[27574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496316, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:36.317718 27736 init.cc:377] Logged in from keytab as kudu/127.21.201.65@KRBTEST.COM (short username kudu)
I20260812 00:58:36.320079 27736 webserver.cc:460] Webserver started at http://127.21.201.65:38139/ using document root <none> and password file <none>
I20260812 00:58:36.321007 27736 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:36.321171 27736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:36.321580 27736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:36.325634 27736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/data/instance:
uuid: "b25ae7c5f9d64e3e8edabf1a91adf08c"
format_stamp: "Formatted at 2026-08-12 00:58:36 on dist-test-slave-7f01"
server_key: "8ae25b973d91dd30885f68ea9e11203b"
server_key_iv: "43285157809d77b291663011c4ba0293"
server_key_version: "encryptionkey@0"
I20260812 00:58:36.326681 27736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance:
uuid: "b25ae7c5f9d64e3e8edabf1a91adf08c"
format_stamp: "Formatted at 2026-08-12 00:58:36 on dist-test-slave-7f01"
server_key: "8ae25b973d91dd30885f68ea9e11203b"
server_key_iv: "43285157809d77b291663011c4ba0293"
server_key_version: "encryptionkey@0"
I20260812 00:58:36.333199 27736 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 00:58:36.337702 27769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:36.339884 27736 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.004s
I20260812 00:58:36.340205 27736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "b25ae7c5f9d64e3e8edabf1a91adf08c"
format_stamp: "Formatted at 2026-08-12 00:58:36 on dist-test-slave-7f01"
server_key: "8ae25b973d91dd30885f68ea9e11203b"
server_key_iv: "43285157809d77b291663011c4ba0293"
server_key_version: "encryptionkey@0"
I20260812 00:58:36.340512 27736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:36.356096 27736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:36.361279 27736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:36.361635 27736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:36.362901 27736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:58:36.364498 27736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:36.364584 27736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:36.364761 27736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:36.364830 27736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:36.406632 27736 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:40327
I20260812 00:58:36.406711 27950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:40327 every 8 connection(s)
I20260812 00:58:36.408502 27736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 00:58:36.414892 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 27736
I20260812 00:58:36.415210 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance
I20260812 00:58:36.415987 22309 external_mini_cluster.cc:1428] Setting key a0c871bd17bbf71aa27542c0b43b0a11
Aug 12 00:58:36 dist-test-slave-7f01 krb5kdc[27574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496316, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:36.434376 27958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:36.412377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:56033 (local address 127.21.201.126:38153)
0812 00:58:36.412890 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:36.412894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:36.413910 (+ 1016us) server_negotiation.cc:407] Connection header received
0812 00:58:36.416249 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:36.416256 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:36.416367 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:36.416517 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:36.418539 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:36.419717 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:36.422129 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:36.422889 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:36.423054 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:36.427517 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:36.427523 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:36.427527 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:36.427565 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:36.430463 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:36.431407 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:36.431413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:36.431417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:36.431615 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:36.432225 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:36.432230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:36.432234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:36.432604 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:36.432777 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:36.433866 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0812 00:58:36.434082 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":287,"threads_started":1}
I20260812 00:58:36.436156 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:36.412918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38153 (local address 127.21.201.65:56033)
0812 00:58:36.413735 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:36.413784 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:36.415894 (+ 2110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:36.416745 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:36.416754 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:36.417505 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:36.418319 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:36.418342 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:36.420065 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:36.420074 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:36.421918 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:36.421939 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:36.423210 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:36.423217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:36.423498 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:36.424384 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:36.424411 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:36.427217 (+ 2806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:36.430661 (+ 3444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:36.430673 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:36.430700 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:36.431267 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:36.431780 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:36.431785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:36.431789 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:36.432024 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:36.432796 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:36.432809 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:36.433391 (+ 582us) client_negotiation.cc:769] Sending connection context
0812 00:58:36.433736 (+ 345us) client_negotiation.cc:240] Negotiation successful
0812 00:58:36.434448 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":322,"threads_started":1}
I20260812 00:58:36.438779 27953 heartbeater.cc:347] Connected to a master server at 127.21.201.126:38153
I20260812 00:58:36.439226 27953 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.21.201.66@KRBTEST.COM; defaulting to no policy
I20260812 00:58:36.440423 27953 heartbeater.cc:511] Master 127.21.201.126:38153 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.66@KRBTEST.COM" created.
I20260812 00:58:36.443635 27641 ts_manager.cc:194] Registered new tserver with Master: b25ae7c5f9d64e3e8edabf1a91adf08c (127.21.201.65:40327)
I20260812 00:58:36.445850 27641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.65@KRBTEST.COM'} at 127.21.201.65:56033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
I20260812 00:58:36.476598 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:0
--local_ip_for_outbound_sockets=127.21.201.66
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38153
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:36841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:36.854820 27965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:36.855353 27965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:36.855581 27965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:36.865887 27965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:36.866070 27965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:36.866212 27965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 00:58:36.877763 27965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:36841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:36.879232 27965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:36.880659 27965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:36.900139 27977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:36.900187 27970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:36.900200 27974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:36.901820 27965 server_base.cc:1034] running on GCE node
I20260812 00:58:36.902572 27965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:36.903861 27965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:36.905218 27965 hybrid_clock.cc:648] HybridClock initialized: now 1786496316905113 us; error 63 us; skew 500 ppm
I20260812 00:58:36.905802 27965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:36 dist-test-slave-7f01 krb5kdc[27574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496316, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:36.910240 27965 init.cc:377] Logged in from keytab as kudu/127.21.201.66@KRBTEST.COM (short username kudu)
I20260812 00:58:36.912761 27965 webserver.cc:460] Webserver started at http://127.21.201.66:42407/ using document root <none> and password file <none>
I20260812 00:58:36.913661 27965 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:36.913781 27965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:36.914090 27965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:36.917491 27965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/data/instance:
uuid: "192e5b8b0d4641ce815027f7050012a9"
format_stamp: "Formatted at 2026-08-12 00:58:36 on dist-test-slave-7f01"
server_key: "ce19bedaeb40b7f120b00b8dfa6bd238"
server_key_iv: "4865b1722ef52708da6d40c4b5b84917"
server_key_version: "encryptionkey@0"
I20260812 00:58:36.918596 27965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance:
uuid: "192e5b8b0d4641ce815027f7050012a9"
format_stamp: "Formatted at 2026-08-12 00:58:36 on dist-test-slave-7f01"
server_key: "ce19bedaeb40b7f120b00b8dfa6bd238"
server_key_iv: "4865b1722ef52708da6d40c4b5b84917"
server_key_version: "encryptionkey@0"
I20260812 00:58:36.925601 27965 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 00:58:36.930714 27986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:36.932798 27965 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 00:58:36.933043 27965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "192e5b8b0d4641ce815027f7050012a9"
format_stamp: "Formatted at 2026-08-12 00:58:36 on dist-test-slave-7f01"
server_key: "ce19bedaeb40b7f120b00b8dfa6bd238"
server_key_iv: "4865b1722ef52708da6d40c4b5b84917"
server_key_version: "encryptionkey@0"
I20260812 00:58:36.933320 27965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:36.947785 27965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:36.952852 27965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:36.953222 27965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:36.954371 27965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:58:36.956111 27965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:36.956247 27965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:36.956394 27965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:36.956462 27965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:36.998039 27965 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:42773
I20260812 00:58:36.998108 28172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:42773 every 8 connection(s)
I20260812 00:58:37.000334 27965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 00:58:37.005749 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 27965
I20260812 00:58:37.005982 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance
I20260812 00:58:37.006582 22309 external_mini_cluster.cc:1428] Setting key e43394f0c16a9ddb0a9a21a7d041f812
Aug 12 00:58:37 dist-test-slave-7f01 krb5kdc[27574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496316, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:37.032912 28177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:37.003562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:50863 (local address 127.21.201.126:38153)
0812 00:58:37.004162 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:37.004170 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:37.005219 (+ 1049us) server_negotiation.cc:407] Connection header received
0812 00:58:37.008042 (+ 2823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:37.008050 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:37.008158 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:37.008294 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:37.011020 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:37.012220 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:37.015847 (+ 3627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:37.016486 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:37.016725 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:37.021932 (+ 5207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:37.021944 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:37.021953 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:37.022030 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:37.027258 (+ 5228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:37.028608 (+ 1350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:37.028617 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:37.028623 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:37.028853 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:37.029549 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:37.029557 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:37.029562 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:37.030389 (+ 827us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:37.030660 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:37.031862 (+ 1202us) server_negotiation.cc:299] Negotiation successful
0812 00:58:37.032346 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":307,"threads_started":1}
I20260812 00:58:37.034297 28179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:37.004201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38153 (local address 127.21.201.66:50863)
0812 00:58:37.005019 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:37.005061 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:37.007552 (+ 2491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:37.008786 (+ 1234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:37.008799 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:37.009781 (+ 982us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:37.010763 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:37.010796 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:37.012761 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:37.012775 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:37.015432 (+ 2657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:37.015462 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:37.016866 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:37.016877 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:37.017286 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:37.018379 (+ 1093us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:37.018426 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:37.021523 (+ 3097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:37.027555 (+ 6032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:37.027568 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:37.027590 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:37.028381 (+ 791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:37.029085 (+ 704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:37.029090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:37.029093 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:37.029386 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:37.030684 (+ 1298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:37.030693 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:37.031274 (+ 581us) client_negotiation.cc:769] Sending connection context
0812 00:58:37.031598 (+ 324us) client_negotiation.cc:240] Negotiation successful
0812 00:58:37.032407 (+ 809us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":321,"threads_started":1}
I20260812 00:58:37.036609 28174 heartbeater.cc:347] Connected to a master server at 127.21.201.126:38153
I20260812 00:58:37.037091 28174 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:37.038237 28174 heartbeater.cc:511] Master 127.21.201.126:38153 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.201.67@KRBTEST.COM; defaulting to no policy
I20260812 00:58:37.040593 27641 ts_manager.cc:194] Registered new tserver with Master: 192e5b8b0d4641ce815027f7050012a9 (127.21.201.66:42773)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.67@KRBTEST.COM" created.
I20260812 00:58:37.041939 27641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.66@KRBTEST.COM'} at 127.21.201.66:50863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
I20260812 00:58:37.072036 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:0
--local_ip_for_outbound_sockets=127.21.201.67
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38153
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:36841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:58:37.450428 27953 heartbeater.cc:503] Master 127.21.201.126:38153 was elected leader, sending a full tablet report...
W20260812 00:58:37.471544 28189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:37.472029 28189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:37.472213 28189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:37.480206 28189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:37.480386 28189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:37.480588 28189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 00:58:37.491242 28189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:36841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:37.492878 28189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:37.494357 28189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:37.510973 28195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:37.511003 28202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:37.511865 28197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:37.511920 28189 server_base.cc:1034] running on GCE node
I20260812 00:58:37.512701 28189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:37.513825 28189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:37.515115 28189 hybrid_clock.cc:648] HybridClock initialized: now 1786496317515033 us; error 78 us; skew 500 ppm
I20260812 00:58:37.515508 28189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:37 dist-test-slave-7f01 krb5kdc[27574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496317, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:37.519641 28189 init.cc:377] Logged in from keytab as kudu/127.21.201.67@KRBTEST.COM (short username kudu)
I20260812 00:58:37.521909 28189 webserver.cc:460] Webserver started at http://127.21.201.67:40185/ using document root <none> and password file <none>
I20260812 00:58:37.522689 28189 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:37.522832 28189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:37.523185 28189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:37.526543 28189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/data/instance:
uuid: "e0d4512ce0ef4cb69c914a6c1bfd8a84"
format_stamp: "Formatted at 2026-08-12 00:58:37 on dist-test-slave-7f01"
server_key: "5ba56062e2b1d73dfc7174ee52bcb912"
server_key_iv: "807bfc475d818d9b7ce60ec04f834738"
server_key_version: "encryptionkey@0"
I20260812 00:58:37.527413 28189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance:
uuid: "e0d4512ce0ef4cb69c914a6c1bfd8a84"
format_stamp: "Formatted at 2026-08-12 00:58:37 on dist-test-slave-7f01"
server_key: "5ba56062e2b1d73dfc7174ee52bcb912"
server_key_iv: "807bfc475d818d9b7ce60ec04f834738"
server_key_version: "encryptionkey@0"
I20260812 00:58:37.533493 28189 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260812 00:58:37.537451 28211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:37.539127 28189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 00:58:37.539326 28189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "e0d4512ce0ef4cb69c914a6c1bfd8a84"
format_stamp: "Formatted at 2026-08-12 00:58:37 on dist-test-slave-7f01"
server_key: "5ba56062e2b1d73dfc7174ee52bcb912"
server_key_iv: "807bfc475d818d9b7ce60ec04f834738"
server_key_version: "encryptionkey@0"
I20260812 00:58:37.539572 28189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:37.553903 28189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:37.558974 28189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:37.559327 28189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:37.560497 28189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:58:37.562112 28189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:37.562227 28189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:37.562388 28189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:37.562464 28189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:37.602051 28189 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:45815
I20260812 00:58:37.602089 28398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:45815 every 8 connection(s)
I20260812 00:58:37.603534 28189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 00:58:37.613224 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 28189
I20260812 00:58:37.613443 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance
I20260812 00:58:37.614173 22309 external_mini_cluster.cc:1428] Setting key 718f4a48c89bfd17d65b5ec478969338
Aug 12 00:58:37 dist-test-slave-7f01 krb5kdc[27574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496317, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:37.629506 28406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:37.606994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:37129 (local address 127.21.201.126:38153)
0812 00:58:37.607546 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:37.607551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:37.608540 (+ 989us) server_negotiation.cc:407] Connection header received
0812 00:58:37.610610 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:37.610616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:37.610718 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:37.610848 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:37.612697 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:37.614052 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:37.616287 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:37.616811 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:37.616959 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:37.621329 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:37.621334 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:37.621338 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:37.621377 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:37.625009 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:37.626157 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:37.626171 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:37.626173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:37.626386 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:37.627025 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:37.627030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:37.627032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:37.627434 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:37.627589 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:37.628780 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0812 00:58:37.629108 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":303,"threads_started":1}
I20260812 00:58:37.630735 28407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:37.607520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38153 (local address 127.21.201.67:37129)
0812 00:58:37.608351 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:37.608393 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:37.610269 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:37.611129 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:37.611138 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:37.611803 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:37.612498 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:37.612516 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:37.614335 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:37.614343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:37.616074 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:37.616095 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:37.617083 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:37.617089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:37.617331 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:37.618253 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:37.618279 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:37.621058 (+ 2779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:37.625275 (+ 4217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:37.625285 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:37.625302 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:37.625852 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:37.626556 (+ 704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:37.626560 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:37.626562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:37.626825 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:37.627647 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:37.627654 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:37.628169 (+ 515us) client_negotiation.cc:769] Sending connection context
0812 00:58:37.628570 (+ 401us) client_negotiation.cc:240] Negotiation successful
0812 00:58:37.629207 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":376,"threads_started":1}
I20260812 00:58:37.633116 28399 heartbeater.cc:347] Connected to a master server at 127.21.201.126:38153
I20260812 00:58:37.633500 28399 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:37.634297 28399 heartbeater.cc:511] Master 127.21.201.126:38153 requested a full tablet report, sending...
I20260812 00:58:37.636305 27641 ts_manager.cc:194] Registered new tserver with Master: e0d4512ce0ef4cb69c914a6c1bfd8a84 (127.21.201.67:45815)
I20260812 00:58:37.637323 27641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.67@KRBTEST.COM'} at 127.21.201.67:37129
I20260812 00:58:37.645499 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 00:58:37 dist-test-slave-7f01 krb5kdc[27574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496317, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 12 00:58:37 dist-test-slave-7f01 krb5kdc[27574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496317, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:37.717777 28406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:37.699132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45662 (local address 127.21.201.126:38153)
0812 00:58:37.699446 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:37.699451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:37.699857 (+ 406us) server_negotiation.cc:407] Connection header received
0812 00:58:37.700113 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:37.700118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:37.700272 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:37.700454 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:37.702212 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:37.703541 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:37.705163 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:37.705670 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:37.705832 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:37.711817 (+ 5985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:37.711833 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:37.711837 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:37.711888 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:37.714829 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:37.715652 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:37.715657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:37.715659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:37.715902 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:37.716387 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:37.716391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:37.716393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:37.716765 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:37.716978 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:37.717309 (+ 331us) server_negotiation.cc:299] Negotiation successful
0812 00:58:37.717522 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
W20260812 00:58:37.718677 27641 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45662
I20260812 00:58:37.720762 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 27736
Aug 12 00:58:37 dist-test-slave-7f01 krb5kdc[27574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496317, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.201.65@KRBTEST.COM
I20260812 00:58:37.741761 28421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:37.722286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55332 (local address 127.21.201.65:40327)
0812 00:58:37.722940 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:37.722949 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:37.723395 (+ 446us) server_negotiation.cc:407] Connection header received
0812 00:58:37.723535 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:37.723542 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:37.723909 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:37.724320 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:37.725868 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:37.727024 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:37.728389 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:37.729015 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:37.729122 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:37.734881 (+ 5759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:37.734899 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:37.734917 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:37.735073 (+ 156us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:37.738450 (+ 3377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:37.739406 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:37.739414 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:37.739417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:37.739567 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:37.739999 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:37.740002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:37.740004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:37.740384 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:37.740622 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:37.741091 (+ 469us) server_negotiation.cc:299] Negotiation successful
0812 00:58:37.741391 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":317,"threads_started":1}
W20260812 00:58:37.743206 27946 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55332
W20260812 00:58:37.744762 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 27736: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 00:58:37.745090 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 27736
I20260812 00:58:37.776245 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 27965
Aug 12 00:58:37 dist-test-slave-7f01 krb5kdc[27574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496317, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.201.66@KRBTEST.COM
I20260812 00:58:37.796607 28425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:37.777510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40466 (local address 127.21.201.66:42773)
0812 00:58:37.778186 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:37.778193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:37.778222 (+ 29us) server_negotiation.cc:407] Connection header received
0812 00:58:37.778300 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:37.778307 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:37.778530 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:37.778678 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:37.780446 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:37.781905 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:37.783580 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:37.784368 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:37.784501 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:37.789460 (+ 4959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:37.789484 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:37.789509 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:37.789571 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:37.793437 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:37.794341 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:37.794350 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:37.794352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:37.794514 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:37.794954 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:37.794958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:37.794960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:37.795344 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:37.795611 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:37.796056 (+ 445us) server_negotiation.cc:299] Negotiation successful
0812 00:58:37.796287 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":381,"threads_started":1}
W20260812 00:58:37.798087 28168 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40466
W20260812 00:58:37.799695 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 27965: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 00:58:37.800143 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 27965
I20260812 00:58:37.841917 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 28189
Aug 12 00:58:37 dist-test-slave-7f01 krb5kdc[27574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496317, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.201.67@KRBTEST.COM
I20260812 00:58:37.864274 28426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:37.843809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43466 (local address 127.21.201.67:45815)
0812 00:58:37.844757 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:37.844766 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:37.844801 (+ 35us) server_negotiation.cc:407] Connection header received
0812 00:58:37.844893 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:37.844900 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:37.845156 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:37.845598 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:37.847606 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:37.849375 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:37.850976 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:37.851621 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:37.851837 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:37.856893 (+ 5056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:37.856910 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:37.856930 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:37.856977 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:37.860590 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:37.861423 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:37.861433 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:37.861435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:37.861574 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:37.861966 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:37.861977 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:37.861980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:37.862361 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:37.862711 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:37.863337 (+ 626us) server_negotiation.cc:299] Negotiation successful
0812 00:58:37.863876 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":537,"threads_started":1}
W20260812 00:58:37.865887 28394 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43466
W20260812 00:58:37.867724 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 28189: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 00:58:37.868065 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 28189
I20260812 00:58:37.910279 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 27592
I20260812 00:58:37.912110 27693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:38.030480 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 27592
2026-08-12T00:58:38Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4613 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:58:38 dist-test-slave-7f01 krb5kdc[28433](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:58:38 dist-test-slave-7f01 krb5kdc[28433](info): set up 2 sockets
Aug 12 00:58:38 dist-test-slave-7f01 krb5kdc[28433](info): commencing operation
krb5kdc: starting...
W20260812 00:58:39.457171 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.367s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 00:58:39 dist-test-slave-7f01 krb5kdc[28433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:58:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:58:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 00:58:39.732185 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:42061
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:38049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:42061
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:40.114143 28454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:40.114502 28454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:40.114606 28454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:40.123196 28454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:58:40.123337 28454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:40.123387 28454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:40.123443 28454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:58:40.123482 28454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:58:40.136005 28454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:38049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:42061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:42061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:40.137454 28454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:40.138794 28454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:40.152678 28464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:40.152740 28468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:40.152866 28466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:40.161949 28454 server_base.cc:1034] running on GCE node
I20260812 00:58:40.163079 28454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:40.165262 28454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:40.166720 28454 hybrid_clock.cc:648] HybridClock initialized: now 1786496320166605 us; error 94 us; skew 500 ppm
I20260812 00:58:40.167374 28454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:40 dist-test-slave-7f01 krb5kdc[28433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496320, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:40.171692 28454 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 00:58:40.174032 28454 webserver.cc:460] Webserver started at http://127.21.201.126:45073/ using document root <none> and password file <none>
I20260812 00:58:40.175040 28454 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:40.175208 28454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:40.175643 28454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:40.179822 28454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "3a2b5218cd1740b0be16df90c3775335"
format_stamp: "Formatted at 2026-08-12 00:58:40 on dist-test-slave-7f01"
server_key: "90543058e6fcb0dcd42d9c0900ca317c"
server_key_iv: "3b86288b0513c111f6c43c27e9f866d9"
server_key_version: "encryptionkey@0"
I20260812 00:58:40.180883 28454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "3a2b5218cd1740b0be16df90c3775335"
format_stamp: "Formatted at 2026-08-12 00:58:40 on dist-test-slave-7f01"
server_key: "90543058e6fcb0dcd42d9c0900ca317c"
server_key_iv: "3b86288b0513c111f6c43c27e9f866d9"
server_key_version: "encryptionkey@0"
I20260812 00:58:40.188454 28454 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 00:58:40.193909 28480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:40.195694 28454 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 00:58:40.196028 28454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "3a2b5218cd1740b0be16df90c3775335"
format_stamp: "Formatted at 2026-08-12 00:58:40 on dist-test-slave-7f01"
server_key: "90543058e6fcb0dcd42d9c0900ca317c"
server_key_iv: "3b86288b0513c111f6c43c27e9f866d9"
server_key_version: "encryptionkey@0"
I20260812 00:58:40.196292 28454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:40.224921 28454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:40.230952 28454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:40.231499 28454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:40.256757 28454 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:42061
I20260812 00:58:40.256780 28568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:42061 every 8 connection(s)
I20260812 00:58:40.258358 28454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 00:58:40.262691 28569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:40.264545 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 28454
I20260812 00:58:40.264906 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 00:58:40.265865 22309 external_mini_cluster.cc:1428] Setting key ba7e1a72ccd69af6fe07b6232ae01b56
I20260812 00:58:40.280932 28569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a2b5218cd1740b0be16df90c3775335: Bootstrap starting.
Aug 12 00:58:40 dist-test-slave-7f01 krb5kdc[28433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:40.285269 28569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a2b5218cd1740b0be16df90c3775335: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:40.286600 28569 log.cc:824] T 00000000000000000000000000000000 P 3a2b5218cd1740b0be16df90c3775335: Log is configured to *not* fsync() on all Append() calls
I20260812 00:58:40.289912 28569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a2b5218cd1740b0be16df90c3775335: No bootstrap required, opened a new log
I20260812 00:58:40.293903 28577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:40.269031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33690 (local address 127.21.201.126:42061)
0812 00:58:40.270205 (+ 1174us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:40.270222 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:40.270277 (+ 55us) server_negotiation.cc:407] Connection header received
0812 00:58:40.272286 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:40.272315 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:40.272901 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:40.273543 (+ 642us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:40.275641 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:40.277267 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:40.279218 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:40.280129 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:40.280292 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:40.285588 (+ 5296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:40.285597 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:40.285614 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:40.285675 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:40.289235 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:40.290113 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:40.290121 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:40.290127 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:40.290296 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:40.290695 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:40.290701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:40.290705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:40.291263 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:40.291606 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:40.292162 (+ 556us) server_negotiation.cc:299] Negotiation successful
0812 00:58:40.292612 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":862,"thread_start_us":578,"threads_started":1}
I20260812 00:58:40.298488 28569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a2b5218cd1740b0be16df90c3775335 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2b5218cd1740b0be16df90c3775335" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42061 } }
I20260812 00:58:40.298928 28569 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a2b5218cd1740b0be16df90c3775335 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:40.299044 28569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a2b5218cd1740b0be16df90c3775335 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a2b5218cd1740b0be16df90c3775335, State: Initialized, Role: FOLLOWER
I20260812 00:58:40.299628 28569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a2b5218cd1740b0be16df90c3775335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2b5218cd1740b0be16df90c3775335" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42061 } }
I20260812 00:58:40.299954 28569 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a2b5218cd1740b0be16df90c3775335 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:58:40.300112 28569 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a2b5218cd1740b0be16df90c3775335 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:58:40.300309 28569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a2b5218cd1740b0be16df90c3775335 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:40.302814 28569 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a2b5218cd1740b0be16df90c3775335 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2b5218cd1740b0be16df90c3775335" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42061 } }
I20260812 00:58:40.303493 28569 leader_election.cc:302] T 00000000000000000000000000000000 P 3a2b5218cd1740b0be16df90c3775335 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a2b5218cd1740b0be16df90c3775335; no voters:
I20260812 00:58:40.304323 28569 leader_election.cc:288] T 00000000000000000000000000000000 P 3a2b5218cd1740b0be16df90c3775335 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:58:40.304756 28579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a2b5218cd1740b0be16df90c3775335 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:58:40.305750 28579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a2b5218cd1740b0be16df90c3775335 [term 1 LEADER]: Becoming Leader. State: Replica: 3a2b5218cd1740b0be16df90c3775335, State: Running, Role: LEADER
I20260812 00:58:40.306578 28579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a2b5218cd1740b0be16df90c3775335 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2b5218cd1740b0be16df90c3775335" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42061 } }
I20260812 00:58:40.306828 28569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a2b5218cd1740b0be16df90c3775335 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:58:40.313041 28582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a2b5218cd1740b0be16df90c3775335 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a2b5218cd1740b0be16df90c3775335" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2b5218cd1740b0be16df90c3775335" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42061 } } }
I20260812 00:58:40.313269 28583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a2b5218cd1740b0be16df90c3775335 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a2b5218cd1740b0be16df90c3775335. Latest consensus state: current_term: 1 leader_uuid: "3a2b5218cd1740b0be16df90c3775335" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2b5218cd1740b0be16df90c3775335" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42061 } } }
I20260812 00:58:40.313772 28583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a2b5218cd1740b0be16df90c3775335 [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:40.313769 28582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a2b5218cd1740b0be16df90c3775335 [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:40.315821 28606 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 00:58:40.317121 28607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:58:40.321834 28607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:58:40.332909 28607 catalog_manager.cc:1347] Generated new cluster ID: 9f41bc62f76d4357af6876fb99cda783
I20260812 00:58:40.333099 28607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:58:40.347033 28607 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:58:40.348596 28607 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:58:40.355903 28607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a2b5218cd1740b0be16df90c3775335: Generated new TSK 0
I20260812 00:58:40.357056 28607 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.201.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
I20260812 00:58:40.432718 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:0
--local_ip_for_outbound_sockets=127.21.201.65
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:42061
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:38049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:40.829643 28620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:40.830049 28620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:40.830229 28620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:40.838301 28620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:40.838483 28620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:40.838671 28620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 00:58:40.849434 28620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:38049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:42061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:40.850981 28620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:40.852542 28620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:40.869380 28632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:40.869181 28628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:40.869475 28634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:40.871201 28620 server_base.cc:1034] running on GCE node
I20260812 00:58:40.872079 28620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:40.873304 28620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:40.874595 28620 hybrid_clock.cc:648] HybridClock initialized: now 1786496320874566 us; error 59 us; skew 500 ppm
I20260812 00:58:40.874984 28620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:40 dist-test-slave-7f01 krb5kdc[28433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496320, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:40.883075 28620 init.cc:377] Logged in from keytab as kudu/127.21.201.65@KRBTEST.COM (short username kudu)
I20260812 00:58:40.885561 28620 webserver.cc:460] Webserver started at http://127.21.201.65:33689/ using document root <none> and password file <none>
I20260812 00:58:40.886691 28620 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:40.886891 28620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:40.887313 28620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:40.891848 28620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-0/data/instance:
uuid: "2969b0165f15494092ffaef7d883c232"
format_stamp: "Formatted at 2026-08-12 00:58:40 on dist-test-slave-7f01"
server_key: "59921a010aa17a1a9a4eb49ceb2e081e"
server_key_iv: "cf26eeecea016b651f01eb5006ad2217"
server_key_version: "encryptionkey@0"
I20260812 00:58:40.892915 28620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance:
uuid: "2969b0165f15494092ffaef7d883c232"
format_stamp: "Formatted at 2026-08-12 00:58:40 on dist-test-slave-7f01"
server_key: "59921a010aa17a1a9a4eb49ceb2e081e"
server_key_iv: "cf26eeecea016b651f01eb5006ad2217"
server_key_version: "encryptionkey@0"
I20260812 00:58:40.900223 28620 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260812 00:58:40.904502 28643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:40.906315 28620 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 00:58:40.906580 28620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "2969b0165f15494092ffaef7d883c232"
format_stamp: "Formatted at 2026-08-12 00:58:40 on dist-test-slave-7f01"
server_key: "59921a010aa17a1a9a4eb49ceb2e081e"
server_key_iv: "cf26eeecea016b651f01eb5006ad2217"
server_key_version: "encryptionkey@0"
I20260812 00:58:40.906863 28620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:40.939546 28620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:40.944826 28620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:40.945180 28620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:40.946411 28620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:58:40.948109 28620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:40.948223 28620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:40.948411 28620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:40.948498 28620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:40.988554 28620 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:41693
I20260812 00:58:40.988616 28829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:41693 every 8 connection(s)
I20260812 00:58:40.989988 28620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 00:58:40.994375 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 28620
I20260812 00:58:40.994822 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance
I20260812 00:58:40.995812 22309 external_mini_cluster.cc:1428] Setting key 73b8302b208b5030b0649eb6c1042234
Aug 12 00:58:41 dist-test-slave-7f01 krb5kdc[28433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496320, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:41.019300 28834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:40.993269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:36131 (local address 127.21.201.126:42061)
0812 00:58:40.994074 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:40.994083 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:40.995241 (+ 1158us) server_negotiation.cc:407] Connection header received
0812 00:58:40.997564 (+ 2323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:40.997570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:40.997666 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:40.997793 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:41.000383 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:41.001487 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:41.004389 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:41.004924 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:41.005102 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:41.010739 (+ 5637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:41.010750 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:41.010758 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:41.010819 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:41.015266 (+ 4447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:41.016327 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:41.016335 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:41.016338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:41.016508 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:41.016957 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:41.016963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:41.016967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:41.017421 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:41.017591 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:41.018451 (+ 860us) server_negotiation.cc:299] Negotiation successful
0812 00:58:41.018757 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":384,"threads_started":1}
I20260812 00:58:41.020385 28836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:40.993989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:42061 (local address 127.21.201.65:36131)
0812 00:58:40.994929 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:40.994981 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:40.997260 (+ 2279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:40.998121 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:40.998135 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:40.999021 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:41.000024 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:41.000060 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:41.001729 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:41.001737 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:41.004118 (+ 2381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:41.004142 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:41.005261 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:41.005272 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:41.005649 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:41.006657 (+ 1008us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:41.006699 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:41.010349 (+ 3650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:41.015480 (+ 5131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:41.015490 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:41.015504 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:41.016165 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:41.016623 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:41.016627 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:41.016629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:41.016819 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:41.017583 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:41.017591 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:41.018002 (+ 411us) client_negotiation.cc:769] Sending connection context
0812 00:58:41.018283 (+ 281us) client_negotiation.cc:240] Negotiation successful
0812 00:58:41.018767 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":409,"threads_started":1}
WARNING: no policy specified for kudu/127.21.201.66@KRBTEST.COM; defaulting to no policy
I20260812 00:58:41.022210 28830 heartbeater.cc:347] Connected to a master server at 127.21.201.126:42061
I20260812 00:58:41.022571 28830 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:41.023412 28830 heartbeater.cc:511] Master 127.21.201.126:42061 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.66@KRBTEST.COM" created.
I20260812 00:58:41.026216 28511 ts_manager.cc:194] Registered new tserver with Master: 2969b0165f15494092ffaef7d883c232 (127.21.201.65:41693)
I20260812 00:58:41.028640 28511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.65@KRBTEST.COM'} at 127.21.201.65:36131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
I20260812 00:58:41.057039 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:0
--local_ip_for_outbound_sockets=127.21.201.66
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:42061
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:38049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:41.426338 28839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:41.426681 28839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:41.426877 28839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:41.434850 28839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:41.434999 28839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:41.435204 28839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 00:58:41.446127 28839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:38049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:42061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:41.447695 28839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:41.449218 28839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:41.464799 28846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:41.464756 28848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:41.464653 28850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:41.467120 28839 server_base.cc:1034] running on GCE node
I20260812 00:58:41.468168 28839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:41.469635 28839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:41.470903 28839 hybrid_clock.cc:648] HybridClock initialized: now 1786496321470812 us; error 84 us; skew 500 ppm
I20260812 00:58:41.471288 28839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:41 dist-test-slave-7f01 krb5kdc[28433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496321, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:41.475816 28839 init.cc:377] Logged in from keytab as kudu/127.21.201.66@KRBTEST.COM (short username kudu)
I20260812 00:58:41.477880 28839 webserver.cc:460] Webserver started at http://127.21.201.66:44367/ using document root <none> and password file <none>
I20260812 00:58:41.478652 28839 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:41.478765 28839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:41.479068 28839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:41.482372 28839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-1/data/instance:
uuid: "96c97ff9d85344fb9518e66da06ef9d5"
format_stamp: "Formatted at 2026-08-12 00:58:41 on dist-test-slave-7f01"
server_key: "1bacdbca113e25a1e714bd07428d8898"
server_key_iv: "61a6fa1048b0ff1540d38e6e07295650"
server_key_version: "encryptionkey@0"
I20260812 00:58:41.483218 28839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance:
uuid: "96c97ff9d85344fb9518e66da06ef9d5"
format_stamp: "Formatted at 2026-08-12 00:58:41 on dist-test-slave-7f01"
server_key: "1bacdbca113e25a1e714bd07428d8898"
server_key_iv: "61a6fa1048b0ff1540d38e6e07295650"
server_key_version: "encryptionkey@0"
I20260812 00:58:41.489687 28839 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260812 00:58:41.493731 28859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:41.495849 28839 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 00:58:41.496224 28839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "96c97ff9d85344fb9518e66da06ef9d5"
format_stamp: "Formatted at 2026-08-12 00:58:41 on dist-test-slave-7f01"
server_key: "1bacdbca113e25a1e714bd07428d8898"
server_key_iv: "61a6fa1048b0ff1540d38e6e07295650"
server_key_version: "encryptionkey@0"
I20260812 00:58:41.496529 28839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:41.510110 28839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:41.515998 28839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:41.516356 28839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:41.517550 28839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:58:41.519112 28839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:41.519227 28839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:41.519387 28839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:41.519465 28839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:41.558717 28839 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:39529
I20260812 00:58:41.558792 29047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:39529 every 8 connection(s)
I20260812 00:58:41.560524 28839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 00:58:41.564394 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 28839
I20260812 00:58:41.564666 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance
I20260812 00:58:41.565572 22309 external_mini_cluster.cc:1428] Setting key 3186f1e03b140f8bcd3e972d68a7a2b2
Aug 12 00:58:41 dist-test-slave-7f01 krb5kdc[28433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496321, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:41.584725 29054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:41.563588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:54173 (local address 127.21.201.126:42061)
0812 00:58:41.564179 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:41.564184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:41.565389 (+ 1205us) server_negotiation.cc:407] Connection header received
0812 00:58:41.567913 (+ 2524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:41.567917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:41.567990 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:41.568126 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:41.569950 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:41.570739 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:41.572506 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:41.572965 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:41.573105 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:41.577098 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:41.577104 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:41.577107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:41.577147 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:41.579994 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:41.581300 (+ 1306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:41.581308 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:41.581311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:41.581557 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:41.582432 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:41.582437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:41.582440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:41.582814 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:41.582954 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:41.584215 (+ 1261us) server_negotiation.cc:299] Negotiation successful
0812 00:58:41.584419 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":338,"threads_started":1}
I20260812 00:58:41.586355 29056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:41.564104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:42061 (local address 127.21.201.66:54173)
0812 00:58:41.565166 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:41.565209 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:41.567407 (+ 2198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:41.568412 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:41.568421 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:41.569120 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:41.569753 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:41.569771 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:41.570930 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:41.570934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:41.572345 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:41.572358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:41.573236 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:41.573245 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:41.573502 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:41.574332 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:41.574363 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:41.576802 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:41.580159 (+ 3357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:41.580173 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:41.580194 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:41.580920 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:41.581736 (+ 816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:41.581743 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:41.581749 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:41.582208 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:41.583023 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:41.583036 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:41.583672 (+ 636us) client_negotiation.cc:769] Sending connection context
0812 00:58:41.584060 (+ 388us) client_negotiation.cc:240] Negotiation successful
0812 00:58:41.584737 (+ 677us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":474,"threads_started":1}
I20260812 00:58:41.588248 29051 heartbeater.cc:347] Connected to a master server at 127.21.201.126:42061
I20260812 00:58:41.588680 29051 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:41.589561 29051 heartbeater.cc:511] Master 127.21.201.126:42061 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.201.67@KRBTEST.COM; defaulting to no policy
I20260812 00:58:41.591341 28511 ts_manager.cc:194] Registered new tserver with Master: 96c97ff9d85344fb9518e66da06ef9d5 (127.21.201.66:39529)
I20260812 00:58:41.592306 28511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.66@KRBTEST.COM'} at 127.21.201.66:54173
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
I20260812 00:58:41.632130 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:0
--local_ip_for_outbound_sockets=127.21.201.67
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:42061
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:38049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:42.025978 29061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:42.026331 29061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:42.026533 29061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260812 00:58:42.032723 28830 heartbeater.cc:503] Master 127.21.201.126:42061 was elected leader, sending a full tablet report...
W20260812 00:58:42.036145 29061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:42.036317 29061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:42.036525 29061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 00:58:42.047585 29061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:38049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:42061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:42.049139 29061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:42.050582 29061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:42.066001 29080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:42.066154 29078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:42.066120 29075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:42.067188 29061 server_base.cc:1034] running on GCE node
I20260812 00:58:42.068017 29061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:42.069235 29061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:42.070451 29061 hybrid_clock.cc:648] HybridClock initialized: now 1786496322070380 us; error 66 us; skew 500 ppm
I20260812 00:58:42.070814 29061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:42 dist-test-slave-7f01 krb5kdc[28433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496322, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:42.074573 29061 init.cc:377] Logged in from keytab as kudu/127.21.201.67@KRBTEST.COM (short username kudu)
I20260812 00:58:42.080066 29061 webserver.cc:460] Webserver started at http://127.21.201.67:43211/ using document root <none> and password file <none>
I20260812 00:58:42.080886 29061 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:42.081018 29061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:42.081308 29061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:42.084774 29061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-2/data/instance:
uuid: "0566137f9c4743f898bd4656fe642579"
format_stamp: "Formatted at 2026-08-12 00:58:42 on dist-test-slave-7f01"
server_key: "11e0cf18523612700a69164d562ff2c8"
server_key_iv: "9b7a7f838e81a0a042b3866549f788bc"
server_key_version: "encryptionkey@0"
I20260812 00:58:42.085752 29061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance:
uuid: "0566137f9c4743f898bd4656fe642579"
format_stamp: "Formatted at 2026-08-12 00:58:42 on dist-test-slave-7f01"
server_key: "11e0cf18523612700a69164d562ff2c8"
server_key_iv: "9b7a7f838e81a0a042b3866549f788bc"
server_key_version: "encryptionkey@0"
I20260812 00:58:42.092221 29061 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 00:58:42.096405 29091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:42.097887 29061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 00:58:42.098119 29061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "0566137f9c4743f898bd4656fe642579"
format_stamp: "Formatted at 2026-08-12 00:58:42 on dist-test-slave-7f01"
server_key: "11e0cf18523612700a69164d562ff2c8"
server_key_iv: "9b7a7f838e81a0a042b3866549f788bc"
server_key_version: "encryptionkey@0"
I20260812 00:58:42.098376 29061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:42.124420 29061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:42.130302 29061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:42.130869 29061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:42.132416 29061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:58:42.134299 29061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:42.134421 29061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:42.134605 29061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:42.134684 29061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:42.177148 29061 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:35963
I20260812 00:58:42.177289 29270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:35963 every 8 connection(s)
I20260812 00:58:42.178660 29061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 00:58:42.182494 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 29061
I20260812 00:58:42.182780 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance
I20260812 00:58:42.183554 22309 external_mini_cluster.cc:1428] Setting key 3bcae532781c385a20433c677c05d8e2
Aug 12 00:58:42 dist-test-slave-7f01 krb5kdc[28433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496322, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:42.204849 29273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:42.181731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:47961 (local address 127.21.201.126:42061)
0812 00:58:42.184190 (+ 2459us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:42.184198 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:42.184226 (+ 28us) server_negotiation.cc:407] Connection header received
0812 00:58:42.185729 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:42.185735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:42.185836 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:42.185960 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:42.187661 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:42.188560 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:42.190203 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:42.190640 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:42.190784 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:42.195710 (+ 4926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:42.195716 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:42.195720 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:42.195925 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:42.199588 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:42.201594 (+ 2006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:42.201600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:42.201603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:42.202014 (+ 411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:42.202664 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:42.202671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:42.202675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:42.203100 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:42.203279 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:42.204148 (+ 869us) server_negotiation.cc:299] Negotiation successful
0812 00:58:42.204423 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2319,"thread_start_us":2177,"threads_started":1}
I20260812 00:58:42.206053 29274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:42.182167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:42061 (local address 127.21.201.67:47961)
0812 00:58:42.183279 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:42.183337 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:42.185441 (+ 2104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:42.186234 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:42.186242 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:42.186838 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:42.187469 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:42.187494 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:42.188738 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:42.188743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:42.190044 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:42.190058 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:42.190905 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:42.190916 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:42.191240 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:42.192331 (+ 1091us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:42.192382 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:42.195402 (+ 3020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:42.200223 (+ 4821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:42.200243 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:42.200277 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:42.201363 (+ 1086us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:42.202197 (+ 834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:42.202203 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:42.202209 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:42.202518 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:42.203241 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:42.203250 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:42.203666 (+ 416us) client_negotiation.cc:769] Sending connection context
0812 00:58:42.204062 (+ 396us) client_negotiation.cc:240] Negotiation successful
0812 00:58:42.204723 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"thread_start_us":518,"threads_started":1}
I20260812 00:58:42.207895 29271 heartbeater.cc:347] Connected to a master server at 127.21.201.126:42061
I20260812 00:58:42.208221 29271 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:42.208992 29271 heartbeater.cc:511] Master 127.21.201.126:42061 requested a full tablet report, sending...
I20260812 00:58:42.211122 28511 ts_manager.cc:194] Registered new tserver with Master: 0566137f9c4743f898bd4656fe642579 (127.21.201.67:35963)
I20260812 00:58:42.212834 28511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.67@KRBTEST.COM'} at 127.21.201.67:47961
I20260812 00:58:42.214581 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 00:58:42 dist-test-slave-7f01 krb5kdc[28433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 00:58:42 dist-test-slave-7f01 krb5kdc[28433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.65@KRBTEST.COM
I20260812 00:58:42.282687 29282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:42.261240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36454 (local address 127.21.201.65:41693)
0812 00:58:42.262027 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:42.262034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:42.262063 (+ 29us) server_negotiation.cc:407] Connection header received
0812 00:58:42.262128 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:42.262134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:42.262371 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:42.262504 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:42.264586 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:42.265901 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:42.267538 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:42.268365 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:42.268518 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:42.274500 (+ 5982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:42.274518 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:42.274535 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:42.274580 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:42.278393 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:42.279450 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:42.279461 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:42.279464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:42.279645 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:42.280314 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:42.280320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:42.280323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:42.280941 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:42.281514 (+ 573us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:42.281856 (+ 342us) server_negotiation.cc:299] Negotiation successful
0812 00:58:42.282332 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":423,"threads_started":1}
Aug 12 00:58:42 dist-test-slave-7f01 krb5kdc[28433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:42.307202 29273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:42.288879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33700 (local address 127.21.201.126:42061)
0812 00:58:42.289187 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:42.289194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:42.289296 (+ 102us) server_negotiation.cc:407] Connection header received
0812 00:58:42.289440 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:42.289445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:42.289552 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:42.289661 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:42.291112 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:42.292064 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:42.293369 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:42.293801 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:42.293901 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:42.299319 (+ 5418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:42.299327 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:42.299332 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:42.299394 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:42.303323 (+ 3929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:42.304439 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:42.304444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:42.304447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:42.304597 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:42.305124 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:42.305128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:42.305130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:42.305555 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:42.305753 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:42.306412 (+ 659us) server_negotiation.cc:299] Negotiation successful
0812 00:58:42.306741 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260812 00:58:42.308341 28511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33700
I20260812 00:58:42.309821 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 28620
I20260812 00:58:42.326962 29282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:42.311412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36458 (local address 127.21.201.65:41693)
0812 00:58:42.311635 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:42.311647 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:42.312015 (+ 368us) server_negotiation.cc:407] Connection header received
0812 00:58:42.312285 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:42.312291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:42.312507 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:42.312642 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:42.314144 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:42.314919 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:42.316836 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:42.317348 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:42.317520 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:42.320539 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:42.320545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:42.320549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:42.320593 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:42.323959 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:42.324772 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:42.324776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:42.324779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:42.324930 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:42.325379 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:42.325384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:42.325386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:42.325689 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:42.325964 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:42.326319 (+ 355us) server_negotiation.cc:299] Negotiation successful
0812 00:58:42.326592 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 00:58:42.327834 28821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:42.418815 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 28620
I20260812 00:58:42.448081 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 28839
Aug 12 00:58:42 dist-test-slave-7f01 krb5kdc[28433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.66@KRBTEST.COM
I20260812 00:58:42.468606 29293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:42.449482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41396 (local address 127.21.201.66:39529)
0812 00:58:42.450285 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:42.450295 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:42.450324 (+ 29us) server_negotiation.cc:407] Connection header received
0812 00:58:42.450414 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:42.450421 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:42.450651 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:42.450886 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:42.452435 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:42.453526 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:42.455419 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:42.456281 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:42.456474 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:42.461551 (+ 5077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:42.461569 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:42.461586 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:42.461632 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:42.465358 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:42.466181 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:42.466189 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:42.466191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:42.466339 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:42.466756 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:42.466761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:42.466783 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:42.467184 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:42.467542 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:42.467908 (+ 366us) server_negotiation.cc:299] Negotiation successful
0812 00:58:42.468214 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":340,"threads_started":1}
I20260812 00:58:42.470463 29039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:42.562273 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 28839
I20260812 00:58:42.592484 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 29061
Aug 12 00:58:42 dist-test-slave-7f01 krb5kdc[28433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.67@KRBTEST.COM
I20260812 00:58:42.614876 29297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:42.593802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60606 (local address 127.21.201.67:35963)
0812 00:58:42.594525 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:42.594533 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:42.594565 (+ 32us) server_negotiation.cc:407] Connection header received
0812 00:58:42.594630 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:42.594637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:42.594860 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:42.595003 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:42.596552 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:42.597570 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:42.599100 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:42.599956 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:42.600071 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:42.605662 (+ 5591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:42.605687 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:42.605717 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:42.605786 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:42.610337 (+ 4551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:42.611884 (+ 1547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:42.611897 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:42.611900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:42.612152 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:42.612741 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:42.612747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:42.612750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:42.613261 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:42.613571 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:42.614087 (+ 516us) server_negotiation.cc:299] Negotiation successful
0812 00:58:42.614364 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":382,"threads_started":1}
I20260812 00:58:42.616748 29258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:42.706467 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 29061
I20260812 00:58:42.730659 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 28454
I20260812 00:58:42.732573 28561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:42.846802 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 28454
2026-08-12T00:58:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4821 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:58:42 dist-test-slave-7f01 krb5kdc[29307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:58:42 dist-test-slave-7f01 krb5kdc[29307](info): set up 2 sockets
Aug 12 00:58:42 dist-test-slave-7f01 krb5kdc[29307](info): commencing operation
krb5kdc: starting...
W20260812 00:58:44.273250 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.362s user 0.006s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 00:58:44 dist-test-slave-7f01 krb5kdc[29307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:58:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:58:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 00:58:44.552749 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:40175
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:33775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:40175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:44.928486 29328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:44.928855 29328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:44.928961 29328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:44.937338 29328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:58:44.937503 29328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:44.937556 29328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:44.937616 29328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:58:44.937668 29328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:58:44.948344 29328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:33775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:40175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:40175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:44.949774 29328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:44.951133 29328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:44.963912 29335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:44.963975 29339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:44.964025 29336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:44.965297 29328 server_base.cc:1034] running on GCE node
I20260812 00:58:44.966666 29328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:44.968830 29328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:44.970220 29328 hybrid_clock.cc:648] HybridClock initialized: now 1786496324970150 us; error 63 us; skew 500 ppm
I20260812 00:58:44.970739 29328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:44 dist-test-slave-7f01 krb5kdc[29307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496324, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:44.974833 29328 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 00:58:44.976567 29328 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:44.976725 29328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:44.977103 29328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:44.980647 29328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "a6c27c7589294cffac2dfb1b633c675f"
format_stamp: "Formatted at 2026-08-12 00:58:44 on dist-test-slave-7f01"
server_key: "4497e1ec83e7c5f7fb38732234dfcdc5"
server_key_iv: "247028741084713ce6bba5c1f7d84fff"
server_key_version: "encryptionkey@0"
I20260812 00:58:44.981559 29328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "a6c27c7589294cffac2dfb1b633c675f"
format_stamp: "Formatted at 2026-08-12 00:58:44 on dist-test-slave-7f01"
server_key: "4497e1ec83e7c5f7fb38732234dfcdc5"
server_key_iv: "247028741084713ce6bba5c1f7d84fff"
server_key_version: "encryptionkey@0"
I20260812 00:58:44.988327 29328 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 00:58:44.993873 29354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:44.995584 29328 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 00:58:44.995900 29328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "a6c27c7589294cffac2dfb1b633c675f"
format_stamp: "Formatted at 2026-08-12 00:58:44 on dist-test-slave-7f01"
server_key: "4497e1ec83e7c5f7fb38732234dfcdc5"
server_key_iv: "247028741084713ce6bba5c1f7d84fff"
server_key_version: "encryptionkey@0"
I20260812 00:58:44.996193 29328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:45.022888 29328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:45.029561 29328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:45.029932 29328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:45.054306 29328 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:40175
I20260812 00:58:45.054327 29445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:40175 every 8 connection(s)
I20260812 00:58:45.055351 29328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 00:58:45.059497 29447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:45.063346 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 29328
I20260812 00:58:45.063632 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 00:58:45.064503 22309 external_mini_cluster.cc:1428] Setting key 6ebdcbc6a9cdefddd11259081ef5e7ef
I20260812 00:58:45.074573 29447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6c27c7589294cffac2dfb1b633c675f: Bootstrap starting.
I20260812 00:58:45.078817 29447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6c27c7589294cffac2dfb1b633c675f: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:45.080359 29447 log.cc:824] T 00000000000000000000000000000000 P a6c27c7589294cffac2dfb1b633c675f: Log is configured to *not* fsync() on all Append() calls
Aug 12 00:58:45 dist-test-slave-7f01 krb5kdc[29307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:45.084868 29447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6c27c7589294cffac2dfb1b633c675f: No bootstrap required, opened a new log
I20260812 00:58:45.095345 29447 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6c27c7589294cffac2dfb1b633c675f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c27c7589294cffac2dfb1b633c675f" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 40175 } }
I20260812 00:58:45.095727 29447 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6c27c7589294cffac2dfb1b633c675f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:45.095801 29454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:45.067051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47344 (local address 127.21.201.126:40175)
0812 00:58:45.068342 (+ 1291us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:45.068362 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:45.068427 (+ 65us) server_negotiation.cc:407] Connection header received
0812 00:58:45.070448 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:45.070482 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:45.071097 (+ 615us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:45.071700 (+ 603us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:45.073704 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:45.075268 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:45.076845 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:45.077560 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:45.077771 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:45.083795 (+ 6024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:45.083812 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:45.083868 (+ 56us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:45.083957 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:45.090062 (+ 6105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:45.091259 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:45.091269 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:45.091279 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:45.091564 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:45.092039 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:45.092042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:45.092044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:45.092669 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:45.092917 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:45.093893 (+ 976us) server_negotiation.cc:299] Negotiation successful
0812 00:58:45.094564 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":959,"thread_start_us":574,"threads_started":1}
I20260812 00:58:45.095897 29447 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6c27c7589294cffac2dfb1b633c675f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6c27c7589294cffac2dfb1b633c675f, State: Initialized, Role: FOLLOWER
I20260812 00:58:45.096520 29447 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6c27c7589294cffac2dfb1b633c675f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c27c7589294cffac2dfb1b633c675f" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 40175 } }
I20260812 00:58:45.096836 29447 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6c27c7589294cffac2dfb1b633c675f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:58:45.096972 29447 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6c27c7589294cffac2dfb1b633c675f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:58:45.097157 29447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6c27c7589294cffac2dfb1b633c675f [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:45.100492 29447 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6c27c7589294cffac2dfb1b633c675f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c27c7589294cffac2dfb1b633c675f" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 40175 } }
I20260812 00:58:45.101074 29447 leader_election.cc:302] T 00000000000000000000000000000000 P a6c27c7589294cffac2dfb1b633c675f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6c27c7589294cffac2dfb1b633c675f; no voters:
I20260812 00:58:45.101754 29447 leader_election.cc:288] T 00000000000000000000000000000000 P a6c27c7589294cffac2dfb1b633c675f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:58:45.101928 29457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6c27c7589294cffac2dfb1b633c675f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:58:45.102694 29457 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6c27c7589294cffac2dfb1b633c675f [term 1 LEADER]: Becoming Leader. State: Replica: a6c27c7589294cffac2dfb1b633c675f, State: Running, Role: LEADER
I20260812 00:58:45.103259 29457 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6c27c7589294cffac2dfb1b633c675f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c27c7589294cffac2dfb1b633c675f" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 40175 } }
I20260812 00:58:45.104056 29447 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6c27c7589294cffac2dfb1b633c675f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:58:45.106141 29458 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6c27c7589294cffac2dfb1b633c675f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6c27c7589294cffac2dfb1b633c675f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c27c7589294cffac2dfb1b633c675f" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 40175 } } }
I20260812 00:58:45.107234 29458 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6c27c7589294cffac2dfb1b633c675f [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:45.106803 29459 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6c27c7589294cffac2dfb1b633c675f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6c27c7589294cffac2dfb1b633c675f. Latest consensus state: current_term: 1 leader_uuid: "a6c27c7589294cffac2dfb1b633c675f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c27c7589294cffac2dfb1b633c675f" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 40175 } } }
I20260812 00:58:45.107853 29459 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6c27c7589294cffac2dfb1b633c675f [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:45.112046 29470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:58:45.116483 29470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:58:45.125749 29470 catalog_manager.cc:1347] Generated new cluster ID: a45a949663cf48f8bcc92f427f028cec
I20260812 00:58:45.125893 29470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:58:45.142829 29470 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:58:45.144604 29470 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:58:45.152522 29470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6c27c7589294cffac2dfb1b633c675f: Generated new TSK 0
I20260812 00:58:45.153746 29470 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.201.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
I20260812 00:58:45.216316 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:0
--local_ip_for_outbound_sockets=127.21.201.65
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:40175
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:33775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:45.587082 29494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:45.587420 29494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:45.587594 29494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:45.595520 29494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:45.595659 29494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:45.595918 29494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 00:58:45.607002 29494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:33775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:40175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:45.608712 29494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:45.610272 29494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:45.624701 29513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:45.624679 29508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:45.624802 29509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:45.625978 29494 server_base.cc:1034] running on GCE node
I20260812 00:58:45.626718 29494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:45.627912 29494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:45.629168 29494 hybrid_clock.cc:648] HybridClock initialized: now 1786496325629103 us; error 65 us; skew 500 ppm
I20260812 00:58:45.629525 29494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:45 dist-test-slave-7f01 krb5kdc[29307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496325, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:45.633538 29494 init.cc:377] Logged in from keytab as kudu/127.21.201.65@KRBTEST.COM (short username kudu)
I20260812 00:58:45.634891 29494 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:45.635001 29494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:45.635389 29494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:45.638773 29494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-0/data/instance:
uuid: "ad50350dbfc945ec9a97f4bcf2c1ad9c"
format_stamp: "Formatted at 2026-08-12 00:58:45 on dist-test-slave-7f01"
server_key: "b621d9dfc1c027ccb8bfdc322c0f0275"
server_key_iv: "f28904d6322c52d6837a7b6abb444e23"
server_key_version: "encryptionkey@0"
I20260812 00:58:45.639788 29494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance:
uuid: "ad50350dbfc945ec9a97f4bcf2c1ad9c"
format_stamp: "Formatted at 2026-08-12 00:58:45 on dist-test-slave-7f01"
server_key: "b621d9dfc1c027ccb8bfdc322c0f0275"
server_key_iv: "f28904d6322c52d6837a7b6abb444e23"
server_key_version: "encryptionkey@0"
I20260812 00:58:45.645495 29494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 00:58:45.649570 29526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:45.651235 29494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 00:58:45.651468 29494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "ad50350dbfc945ec9a97f4bcf2c1ad9c"
format_stamp: "Formatted at 2026-08-12 00:58:45 on dist-test-slave-7f01"
server_key: "b621d9dfc1c027ccb8bfdc322c0f0275"
server_key_iv: "f28904d6322c52d6837a7b6abb444e23"
server_key_version: "encryptionkey@0"
I20260812 00:58:45.651732 29494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:45.667248 29494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:45.672220 29494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:45.672547 29494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:45.673630 29494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:58:45.675133 29494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:45.675266 29494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:45.675405 29494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:45.675503 29494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:45.716430 29494 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:38725
I20260812 00:58:45.716498 29701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:38725 every 8 connection(s)
I20260812 00:58:45.717404 29494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 00:58:45.723243 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 29494
I20260812 00:58:45.723526 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance
I20260812 00:58:45.724288 22309 external_mini_cluster.cc:1428] Setting key 9c0bf3f5ebea0de69295f6180625285f
Aug 12 00:58:45 dist-test-slave-7f01 krb5kdc[29307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496325, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:45.743254 29706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:45.720889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:45855 (local address 127.21.201.126:40175)
0812 00:58:45.721384 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:45.721389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:45.723125 (+ 1736us) server_negotiation.cc:407] Connection header received
0812 00:58:45.726165 (+ 3040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:45.726171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:45.726244 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:45.726362 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:45.728384 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:45.729144 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:45.730814 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:45.731247 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:45.731392 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:45.736793 (+ 5401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:45.736799 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:45.736803 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:45.736844 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:45.739590 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:45.740558 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:45.740569 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:45.740571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:45.740697 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:45.741180 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:45.741184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:45.741186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:45.741496 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:45.741663 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:45.742467 (+ 804us) server_negotiation.cc:299] Negotiation successful
0812 00:58:45.742849 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":268,"threads_started":1}
I20260812 00:58:45.744511 29708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:45.721680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:40175 (local address 127.21.201.65:45855)
0812 00:58:45.722758 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:45.722866 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:45.725752 (+ 2886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:45.726743 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:45.726756 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:45.727448 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:45.728178 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:45.728198 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:45.729327 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:45.729331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:45.730637 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:45.730650 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:45.731517 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:45.731524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:45.731822 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:45.732770 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:45.732813 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:45.736437 (+ 3624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:45.739828 (+ 3391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:45.739839 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:45.739872 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:45.740406 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:45.740820 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:45.740826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:45.740830 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:45.741047 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:45.741635 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:45.741643 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:45.742040 (+ 397us) client_negotiation.cc:769] Sending connection context
0812 00:58:45.742343 (+ 303us) client_negotiation.cc:240] Negotiation successful
0812 00:58:45.743136 (+ 793us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":475,"threads_started":1}
I20260812 00:58:45.746409 29702 heartbeater.cc:347] Connected to a master server at 127.21.201.126:40175
I20260812 00:58:45.746814 29702 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:45.747807 29702 heartbeater.cc:511] Master 127.21.201.126:40175 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.201.66@KRBTEST.COM; defaulting to no policy
I20260812 00:58:45.750983 29383 ts_manager.cc:194] Registered new tserver with Master: ad50350dbfc945ec9a97f4bcf2c1ad9c (127.21.201.65:38725)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.66@KRBTEST.COM" created.
I20260812 00:58:45.753762 29383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.65@KRBTEST.COM'} at 127.21.201.65:45855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
I20260812 00:58:45.784816 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:0
--local_ip_for_outbound_sockets=127.21.201.66
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:40175
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:33775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:46.176501 29714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:46.176911 29714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:46.177088 29714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:46.184995 29714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:46.185199 29714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:46.185385 29714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 00:58:46.197432 29714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:33775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:40175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:46.198989 29714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:46.200569 29714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:46.217954 29727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:46.217965 29729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:46.218134 29725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:46.218925 29714 server_base.cc:1034] running on GCE node
I20260812 00:58:46.219676 29714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:46.220852 29714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:46.222111 29714 hybrid_clock.cc:648] HybridClock initialized: now 1786496326222019 us; error 61 us; skew 500 ppm
I20260812 00:58:46.222506 29714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:46 dist-test-slave-7f01 krb5kdc[29307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496326, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:46.226609 29714 init.cc:377] Logged in from keytab as kudu/127.21.201.66@KRBTEST.COM (short username kudu)
I20260812 00:58:46.228111 29714 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:46.228217 29714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:46.228612 29714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:46.231915 29714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-1/data/instance:
uuid: "81206e5a82194f4cb5f1db23c7e73b84"
format_stamp: "Formatted at 2026-08-12 00:58:46 on dist-test-slave-7f01"
server_key: "cd52772b71885cc81fd263aba587b3fe"
server_key_iv: "1bb9b1ab179f8efeccdda83baf2348e9"
server_key_version: "encryptionkey@0"
I20260812 00:58:46.232894 29714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance:
uuid: "81206e5a82194f4cb5f1db23c7e73b84"
format_stamp: "Formatted at 2026-08-12 00:58:46 on dist-test-slave-7f01"
server_key: "cd52772b71885cc81fd263aba587b3fe"
server_key_iv: "1bb9b1ab179f8efeccdda83baf2348e9"
server_key_version: "encryptionkey@0"
I20260812 00:58:46.239503 29714 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 00:58:46.245028 29738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:46.247020 29714 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 00:58:46.247279 29714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "81206e5a82194f4cb5f1db23c7e73b84"
format_stamp: "Formatted at 2026-08-12 00:58:46 on dist-test-slave-7f01"
server_key: "cd52772b71885cc81fd263aba587b3fe"
server_key_iv: "1bb9b1ab179f8efeccdda83baf2348e9"
server_key_version: "encryptionkey@0"
I20260812 00:58:46.247579 29714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:46.262990 29714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:46.268834 29714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:46.269207 29714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:46.270474 29714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:58:46.272172 29714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:46.272311 29714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:46.272468 29714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:46.272572 29714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:46.311955 29714 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:33785
I20260812 00:58:46.312006 29926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:33785 every 8 connection(s)
I20260812 00:58:46.312938 29714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 00:58:46.320581 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 29714
I20260812 00:58:46.320864 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance
I20260812 00:58:46.321516 22309 external_mini_cluster.cc:1428] Setting key e7785d015ba276e235f849818fad99d4
Aug 12 00:58:46 dist-test-slave-7f01 krb5kdc[29307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496326, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:46.341804 29931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:46.315800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:48063 (local address 127.21.201.126:40175)
0812 00:58:46.316508 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:46.316515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:46.317299 (+ 784us) server_negotiation.cc:407] Connection header received
0812 00:58:46.319288 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:46.319294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:46.319390 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:46.319546 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:46.322687 (+ 3141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:46.323658 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:46.326252 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:46.326778 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:46.326911 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:46.332252 (+ 5341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:46.332258 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:46.332262 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:46.332304 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:46.336306 (+ 4002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:46.337702 (+ 1396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:46.337711 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:46.337717 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:46.337912 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:46.338678 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:46.338687 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:46.338693 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:46.339133 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:46.339309 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:46.340678 (+ 1369us) server_negotiation.cc:299] Negotiation successful
0812 00:58:46.340996 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":298,"threads_started":1}
I20260812 00:58:46.342726 29934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:46.316199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:40175 (local address 127.21.201.66:48063)
0812 00:58:46.317083 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:46.317131 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:46.318952 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:46.320156 (+ 1204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:46.320176 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:46.321219 (+ 1043us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:46.322372 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:46.322409 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:46.323889 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:46.323894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:46.326037 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:46.326058 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:46.327046 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:46.327057 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:46.327414 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:46.328332 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:46.328369 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:46.331868 (+ 3499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:46.336531 (+ 4663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:46.336541 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:46.336557 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:46.337452 (+ 895us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:46.338171 (+ 719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:46.338176 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:46.338179 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:46.338500 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:46.339334 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:46.339344 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:46.339890 (+ 546us) client_negotiation.cc:769] Sending connection context
0812 00:58:46.340509 (+ 619us) client_negotiation.cc:240] Negotiation successful
0812 00:58:46.341132 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":377,"threads_started":1}
I20260812 00:58:46.345165 29927 heartbeater.cc:347] Connected to a master server at 127.21.201.126:40175
I20260812 00:58:46.345573 29927 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:46.346364 29927 heartbeater.cc:511] Master 127.21.201.126:40175 requested a full tablet report, sending...
I20260812 00:58:46.348188 29383 ts_manager.cc:194] Registered new tserver with Master: 81206e5a82194f4cb5f1db23c7e73b84 (127.21.201.66:33785)
WARNING: no policy specified for kudu/127.21.201.67@KRBTEST.COM; defaulting to no policy
I20260812 00:58:46.349138 29383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.66@KRBTEST.COM'} at 127.21.201.66:48063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
I20260812 00:58:46.384168 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:0
--local_ip_for_outbound_sockets=127.21.201.67
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:40175
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:33775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:58:46.758533 29702 heartbeater.cc:503] Master 127.21.201.126:40175 was elected leader, sending a full tablet report...
W20260812 00:58:46.774958 29941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:46.775513 29941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:46.775635 29941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:46.784659 29941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:46.784811 29941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:46.784946 29941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 00:58:46.795878 29941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:33775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:40175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:46.797326 29941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:46.798728 29941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:46.814441 29947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:46.814519 29949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:46.814541 29952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:46.817842 29941 server_base.cc:1034] running on GCE node
I20260812 00:58:46.818957 29941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:46.820603 29941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:46.821920 29941 hybrid_clock.cc:648] HybridClock initialized: now 1786496326821855 us; error 63 us; skew 500 ppm
I20260812 00:58:46.822376 29941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:46 dist-test-slave-7f01 krb5kdc[29307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496326, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:46.826124 29941 init.cc:377] Logged in from keytab as kudu/127.21.201.67@KRBTEST.COM (short username kudu)
I20260812 00:58:46.827509 29941 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:46.827608 29941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:46.827989 29941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:46.831274 29941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-2/data/instance:
uuid: "773393c70b414223b943f6e7be481471"
format_stamp: "Formatted at 2026-08-12 00:58:46 on dist-test-slave-7f01"
server_key: "9a59400adf83db779f5fe239f49af6b7"
server_key_iv: "a694732cf371b9ce2d1524911098201b"
server_key_version: "encryptionkey@0"
I20260812 00:58:46.832311 29941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance:
uuid: "773393c70b414223b943f6e7be481471"
format_stamp: "Formatted at 2026-08-12 00:58:46 on dist-test-slave-7f01"
server_key: "9a59400adf83db779f5fe239f49af6b7"
server_key_iv: "a694732cf371b9ce2d1524911098201b"
server_key_version: "encryptionkey@0"
I20260812 00:58:46.838428 29941 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 00:58:46.842597 29962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:46.844403 29941 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 00:58:46.844626 29941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "773393c70b414223b943f6e7be481471"
format_stamp: "Formatted at 2026-08-12 00:58:46 on dist-test-slave-7f01"
server_key: "9a59400adf83db779f5fe239f49af6b7"
server_key_iv: "a694732cf371b9ce2d1524911098201b"
server_key_version: "encryptionkey@0"
I20260812 00:58:46.844871 29941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:46.864933 29941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:46.870391 29941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:46.870841 29941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:46.872136 29941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:58:46.873759 29941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:46.873855 29941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:46.874037 29941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:46.874153 29941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:46.913292 29941 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:34301
I20260812 00:58:46.913336 30152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:34301 every 8 connection(s)
I20260812 00:58:46.914268 29941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 00:58:46.915508 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 29941
I20260812 00:58:46.915726 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance
I20260812 00:58:46.916235 22309 external_mini_cluster.cc:1428] Setting key b0736a20f5a9f15db575c813deb0dc9d
Aug 12 00:58:46 dist-test-slave-7f01 krb5kdc[29307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496326, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:46.939474 30158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:46.917599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:38253 (local address 127.21.201.126:40175)
0812 00:58:46.918296 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:46.918304 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:46.919527 (+ 1223us) server_negotiation.cc:407] Connection header received
0812 00:58:46.922131 (+ 2604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:46.922137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:46.922272 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:46.922398 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:46.924365 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:46.925224 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:46.926990 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:46.927467 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:46.927576 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:46.931490 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:46.931498 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:46.931502 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:46.931552 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:46.934967 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:46.935934 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:46.935938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:46.935941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:46.936326 (+ 385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:46.937192 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:46.937196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:46.937198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:46.937568 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:46.937714 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:46.938701 (+ 987us) server_negotiation.cc:299] Negotiation successful
0812 00:58:46.939000 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":365,"threads_started":1}
I20260812 00:58:46.940284 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:46.918058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:40175 (local address 127.21.201.67:38253)
0812 00:58:46.919275 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:46.919376 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:46.921740 (+ 2364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:46.922689 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:46.922699 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:46.923348 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:46.924120 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:46.924147 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:46.925391 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:46.925395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:46.926809 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:46.926822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:46.927704 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:46.927710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:46.928000 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:46.928758 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:46.928783 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:46.931183 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:46.935175 (+ 3992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:46.935184 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:46.935201 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:46.935723 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:46.936576 (+ 853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:46.936582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:46.936586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:46.936963 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:46.937753 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:46.937762 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:46.938191 (+ 429us) client_negotiation.cc:769] Sending connection context
0812 00:58:46.938472 (+ 281us) client_negotiation.cc:240] Negotiation successful
0812 00:58:46.938967 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"thread_start_us":481,"threads_started":1}
I20260812 00:58:46.942319 30156 heartbeater.cc:347] Connected to a master server at 127.21.201.126:40175
I20260812 00:58:46.942696 30156 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:46.943496 30156 heartbeater.cc:511] Master 127.21.201.126:40175 requested a full tablet report, sending...
I20260812 00:58:46.945622 29383 ts_manager.cc:194] Registered new tserver with Master: 773393c70b414223b943f6e7be481471 (127.21.201.67:34301)
I20260812 00:58:46.946826 29383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.67@KRBTEST.COM'} at 127.21.201.67:38253
I20260812 00:58:46.948087 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:58:46.970469 30158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:46.954705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47350 (local address 127.21.201.126:40175)
0812 00:58:46.954957 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:46.954966 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:46.955559 (+ 593us) server_negotiation.cc:407] Connection header received
0812 00:58:46.955830 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:46.955834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:46.955907 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:46.956052 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:46.957719 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:46.958459 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:46.959785 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:46.960212 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:46.960360 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:46.963074 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:46.963079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:46.963083 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:46.963119 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:46.966751 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:46.967806 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:46.967814 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:46.967822 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:46.968000 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:46.968687 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:46.968695 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:46.968699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:46.969158 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:46.969442 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:46.969831 (+ 389us) server_negotiation.cc:299] Negotiation successful
0812 00:58:46.970129 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 00:58:46.976788 29381 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:58:46.984476 29381 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:58:47.017521 30176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:46.998743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:34301 (local address 127.0.0.1:55190)
0812 00:58:47.002763 (+ 4020us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:47.002809 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:47.003143 (+ 334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:47.004427 (+ 1284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:47.004436 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:47.004477 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:58:47.005027 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:47.005045 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.008861 (+ 3816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.008870 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:47.013522 (+ 4652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:47.013550 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.016470 (+ 2920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.016480 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:47.016709 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:47.016771 (+ 62us) client_negotiation.cc:769] Sending connection context
0812 00:58:47.016917 (+ 146us) client_negotiation.cc:240] Negotiation successful
0812 00:58:47.017135 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2552,"thread_start_us":15048,"threads_started":1}
I20260812 00:58:47.017575 30177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:46.998987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55190 (local address 127.21.201.67:34301)
0812 00:58:47.000062 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:47.000075 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:47.002942 (+ 2867us) server_negotiation.cc:407] Connection header received
0812 00:58:47.003421 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:47.003430 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:47.003787 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:47.003991 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:58:47.006597 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.008608 (+ 2011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.013798 (+ 5190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.015773 (+ 1975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.015920 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:47.016212 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:47.017009 (+ 797us) server_negotiation.cc:299] Negotiation successful
0812 00:58:47.017198 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":846,"thread_start_us":642,"threads_started":1}
I20260812 00:58:47.022311 30042 tablet_service.cc:1467] Processing CreateTablet for tablet 4156093e86a342b89f997da9f78eb3bf (DEFAULT_TABLE table=test-table [id=3257c3814c88459980824bbc24008d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:47.023511 30175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:46.999070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:33785 (local address 127.0.0.1:35082)
0812 00:58:47.013490 (+ 14420us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:47.013512 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:47.013679 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:47.015434 (+ 1755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:47.015440 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:47.015468 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:58:47.016051 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:47.016065 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.018051 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.018057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:47.020533 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:47.020547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.022622 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.022629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:47.022813 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:47.022847 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 00:58:47.022944 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 00:58:47.023101 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14242,"thread_start_us":2066,"threads_started":1}
I20260812 00:58:47.023595 30179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:46.999940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35082 (local address 127.21.201.66:33785)
0812 00:58:47.014478 (+ 14538us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:47.014486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:47.014518 (+ 32us) server_negotiation.cc:407] Connection header received
0812 00:58:47.014636 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:47.014647 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:47.014942 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:47.015171 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:58:47.016280 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.017902 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.020710 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.022092 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.022255 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:47.022467 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:47.023041 (+ 574us) server_negotiation.cc:299] Negotiation successful
0812 00:58:47.023228 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14357,"thread_start_us":14734,"threads_started":1}
I20260812 00:58:47.024457 30180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:47.002028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:38725 (local address 127.0.0.1:34198)
0812 00:58:47.014376 (+ 12348us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:47.014409 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:47.014666 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:47.015818 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:47.015823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:47.015852 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:58:47.016394 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:47.016405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.018241 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.018248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:47.021329 (+ 3081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:47.021345 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.023571 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.023581 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:47.023807 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:47.023845 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 00:58:47.023943 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 00:58:47.024082 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12029,"mutex_wait_us":770,"thread_start_us":11836,"threads_started":1}
I20260812 00:58:47.024485 30042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4156093e86a342b89f997da9f78eb3bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:47.024595 30181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:47.002691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34198 (local address 127.21.201.65:38725)
0812 00:58:47.013620 (+ 10929us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:47.013630 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:47.014609 (+ 979us) server_negotiation.cc:407] Connection header received
0812 00:58:47.015070 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:47.015081 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:47.015312 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:47.015508 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:58:47.016592 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.018079 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.021520 (+ 3441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.023028 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.023154 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:47.023397 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:47.024043 (+ 646us) server_negotiation.cc:299] Negotiation successful
0812 00:58:47.024223 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10708,"thread_start_us":10529,"threads_started":1}
I20260812 00:58:47.027410 29822 tablet_service.cc:1467] Processing CreateTablet for tablet 4156093e86a342b89f997da9f78eb3bf (DEFAULT_TABLE table=test-table [id=3257c3814c88459980824bbc24008d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:47.028733 29822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4156093e86a342b89f997da9f78eb3bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:47.028343 29595 tablet_service.cc:1467] Processing CreateTablet for tablet 4156093e86a342b89f997da9f78eb3bf (DEFAULT_TABLE table=test-table [id=3257c3814c88459980824bbc24008d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:47.030069 29595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4156093e86a342b89f997da9f78eb3bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:47.042390 30184 tablet_bootstrap.cc:492] T 4156093e86a342b89f997da9f78eb3bf P 773393c70b414223b943f6e7be481471: Bootstrap starting.
I20260812 00:58:47.045297 30185 tablet_bootstrap.cc:492] T 4156093e86a342b89f997da9f78eb3bf P 81206e5a82194f4cb5f1db23c7e73b84: Bootstrap starting.
I20260812 00:58:47.046999 30184 tablet_bootstrap.cc:654] T 4156093e86a342b89f997da9f78eb3bf P 773393c70b414223b943f6e7be481471: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:47.047879 30187 tablet_bootstrap.cc:492] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c: Bootstrap starting.
I20260812 00:58:47.049203 30184 log.cc:824] T 4156093e86a342b89f997da9f78eb3bf P 773393c70b414223b943f6e7be481471: Log is configured to *not* fsync() on all Append() calls
I20260812 00:58:47.050509 30185 tablet_bootstrap.cc:654] T 4156093e86a342b89f997da9f78eb3bf P 81206e5a82194f4cb5f1db23c7e73b84: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:47.052368 30185 log.cc:824] T 4156093e86a342b89f997da9f78eb3bf P 81206e5a82194f4cb5f1db23c7e73b84: Log is configured to *not* fsync() on all Append() calls
I20260812 00:58:47.053336 30187 tablet_bootstrap.cc:654] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:47.055919 30187 log.cc:824] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c: Log is configured to *not* fsync() on all Append() calls
I20260812 00:58:47.060177 30184 tablet_bootstrap.cc:492] T 4156093e86a342b89f997da9f78eb3bf P 773393c70b414223b943f6e7be481471: No bootstrap required, opened a new log
I20260812 00:58:47.060177 30185 tablet_bootstrap.cc:492] T 4156093e86a342b89f997da9f78eb3bf P 81206e5a82194f4cb5f1db23c7e73b84: No bootstrap required, opened a new log
I20260812 00:58:47.060555 30184 ts_tablet_manager.cc:1397] T 4156093e86a342b89f997da9f78eb3bf P 773393c70b414223b943f6e7be481471: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.008s
I20260812 00:58:47.061339 30185 ts_tablet_manager.cc:1397] T 4156093e86a342b89f997da9f78eb3bf P 81206e5a82194f4cb5f1db23c7e73b84: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.007s
I20260812 00:58:47.070637 30187 tablet_bootstrap.cc:492] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c: No bootstrap required, opened a new log
I20260812 00:58:47.071041 30187 ts_tablet_manager.cc:1397] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.000s
I20260812 00:58:47.073061 30185 raft_consensus.cc:348] T 4156093e86a342b89f997da9f78eb3bf P 81206e5a82194f4cb5f1db23c7e73b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773393c70b414223b943f6e7be481471" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34301 } } peers { permanent_uuid: "81206e5a82194f4cb5f1db23c7e73b84" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33785 } } peers { permanent_uuid: "ad50350dbfc945ec9a97f4bcf2c1ad9c" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 38725 } }
I20260812 00:58:47.074074 30185 raft_consensus.cc:374] T 4156093e86a342b89f997da9f78eb3bf P 81206e5a82194f4cb5f1db23c7e73b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:47.073654 30184 raft_consensus.cc:348] T 4156093e86a342b89f997da9f78eb3bf P 773393c70b414223b943f6e7be481471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773393c70b414223b943f6e7be481471" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34301 } } peers { permanent_uuid: "81206e5a82194f4cb5f1db23c7e73b84" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33785 } } peers { permanent_uuid: "ad50350dbfc945ec9a97f4bcf2c1ad9c" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 38725 } }
I20260812 00:58:47.074215 30185 raft_consensus.cc:729] T 4156093e86a342b89f997da9f78eb3bf P 81206e5a82194f4cb5f1db23c7e73b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81206e5a82194f4cb5f1db23c7e73b84, State: Initialized, Role: FOLLOWER
I20260812 00:58:47.074368 30184 raft_consensus.cc:374] T 4156093e86a342b89f997da9f78eb3bf P 773393c70b414223b943f6e7be481471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:47.074481 30184 raft_consensus.cc:729] T 4156093e86a342b89f997da9f78eb3bf P 773393c70b414223b943f6e7be481471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 773393c70b414223b943f6e7be481471, State: Initialized, Role: FOLLOWER
I20260812 00:58:47.075135 30184 consensus_queue.cc:260] T 4156093e86a342b89f997da9f78eb3bf P 773393c70b414223b943f6e7be481471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773393c70b414223b943f6e7be481471" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34301 } } peers { permanent_uuid: "81206e5a82194f4cb5f1db23c7e73b84" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33785 } } peers { permanent_uuid: "ad50350dbfc945ec9a97f4bcf2c1ad9c" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 38725 } }
I20260812 00:58:47.075135 30185 consensus_queue.cc:260] T 4156093e86a342b89f997da9f78eb3bf P 81206e5a82194f4cb5f1db23c7e73b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773393c70b414223b943f6e7be481471" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34301 } } peers { permanent_uuid: "81206e5a82194f4cb5f1db23c7e73b84" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33785 } } peers { permanent_uuid: "ad50350dbfc945ec9a97f4bcf2c1ad9c" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 38725 } }
I20260812 00:58:47.076545 30156 heartbeater.cc:503] Master 127.21.201.126:40175 was elected leader, sending a full tablet report...
I20260812 00:58:47.076924 29927 heartbeater.cc:503] Master 127.21.201.126:40175 was elected leader, sending a full tablet report...
I20260812 00:58:47.077342 30184 ts_tablet_manager.cc:1428] T 4156093e86a342b89f997da9f78eb3bf P 773393c70b414223b943f6e7be481471: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
I20260812 00:58:47.078819 30185 ts_tablet_manager.cc:1428] T 4156093e86a342b89f997da9f78eb3bf P 81206e5a82194f4cb5f1db23c7e73b84: Time spent starting tablet: real 0.017s user 0.014s sys 0.001s
I20260812 00:58:47.080914 30187 raft_consensus.cc:348] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773393c70b414223b943f6e7be481471" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34301 } } peers { permanent_uuid: "81206e5a82194f4cb5f1db23c7e73b84" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33785 } } peers { permanent_uuid: "ad50350dbfc945ec9a97f4bcf2c1ad9c" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 38725 } }
I20260812 00:58:47.081563 30187 raft_consensus.cc:374] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:47.081735 30187 raft_consensus.cc:729] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad50350dbfc945ec9a97f4bcf2c1ad9c, State: Initialized, Role: FOLLOWER
I20260812 00:58:47.082496 30187 consensus_queue.cc:260] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773393c70b414223b943f6e7be481471" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34301 } } peers { permanent_uuid: "81206e5a82194f4cb5f1db23c7e73b84" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33785 } } peers { permanent_uuid: "ad50350dbfc945ec9a97f4bcf2c1ad9c" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 38725 } }
I20260812 00:58:47.084969 30187 ts_tablet_manager.cc:1428] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260812 00:58:47.166159 30157 tablet.cc:2369] T 4156093e86a342b89f997da9f78eb3bf P 773393c70b414223b943f6e7be481471: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:58:47.220645 29703 tablet.cc:2369] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:58:47.290572 30204 raft_consensus.cc:482] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:58:47.291075 30204 raft_consensus.cc:504] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773393c70b414223b943f6e7be481471" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34301 } } peers { permanent_uuid: "81206e5a82194f4cb5f1db23c7e73b84" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33785 } } peers { permanent_uuid: "ad50350dbfc945ec9a97f4bcf2c1ad9c" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 38725 } }
I20260812 00:58:47.293579 30204 leader_election.cc:288] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 773393c70b414223b943f6e7be481471 (127.21.201.67:34301), 81206e5a82194f4cb5f1db23c7e73b84 (127.21.201.66:33785)
I20260812 00:58:47.303542 30177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:47.294201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:46043 (local address 127.21.201.67:34301)
0812 00:58:47.294448 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:47.294455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:47.295446 (+ 991us) server_negotiation.cc:407] Connection header received
0812 00:58:47.295805 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:47.295811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:47.295904 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:47.296008 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:58:47.297135 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.298421 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.301546 (+ 3125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.302424 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.302656 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:47.302738 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:47.303079 (+ 341us) server_negotiation.cc:299] Negotiation successful
0812 00:58:47.303232 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 00:58:47.303539 30179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:47.294206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:52869 (local address 127.21.201.66:33785)
0812 00:58:47.294448 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:47.294455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:47.295535 (+ 1080us) server_negotiation.cc:407] Connection header received
0812 00:58:47.295684 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:47.295690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:47.295818 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:47.295951 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:58:47.297135 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.298421 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.301456 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.302371 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.302554 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:47.302656 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:47.303089 (+ 433us) server_negotiation.cc:299] Negotiation successful
0812 00:58:47.303235 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 00:58:47.303813 30210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:47.294066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:33785 (local address 127.21.201.65:52869)
0812 00:58:47.295217 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:47.295246 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:47.295476 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:47.296111 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:47.296117 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:47.296176 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:58:47.296838 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:47.296861 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.298663 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.298672 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:47.301218 (+ 2546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:47.301240 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.302660 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.302667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:47.302889 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:47.302923 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 00:58:47.303044 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 00:58:47.303233 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"mutex_wait_us":233,"spinlock_wait_cycles":33664,"thread_start_us":603,"threads_started":1}
I20260812 00:58:47.304158 30211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:47.294177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:34301 (local address 127.21.201.65:46043)
0812 00:58:47.295245 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:47.295260 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:47.295515 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:47.296111 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:47.296117 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:47.296173 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:58:47.296869 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:47.296878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.298743 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.298749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:47.301234 (+ 2485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:47.301253 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.302728 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.302733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:47.302902 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:47.302927 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 00:58:47.303047 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 00:58:47.303227 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"mutex_wait_us":72,"spinlock_wait_cycles":56448,"thread_start_us":498,"threads_started":1}
I20260812 00:58:47.305055 29858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4156093e86a342b89f997da9f78eb3bf" candidate_uuid: "ad50350dbfc945ec9a97f4bcf2c1ad9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81206e5a82194f4cb5f1db23c7e73b84" is_pre_election: true
I20260812 00:58:47.305227 30068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4156093e86a342b89f997da9f78eb3bf" candidate_uuid: "ad50350dbfc945ec9a97f4bcf2c1ad9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "773393c70b414223b943f6e7be481471" is_pre_election: true
I20260812 00:58:47.305732 29858 raft_consensus.cc:2454] T 4156093e86a342b89f997da9f78eb3bf P 81206e5a82194f4cb5f1db23c7e73b84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad50350dbfc945ec9a97f4bcf2c1ad9c in term 0.
I20260812 00:58:47.305804 30068 raft_consensus.cc:2454] T 4156093e86a342b89f997da9f78eb3bf P 773393c70b414223b943f6e7be481471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad50350dbfc945ec9a97f4bcf2c1ad9c in term 0.
I20260812 00:58:47.306828 29527 leader_election.cc:302] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81206e5a82194f4cb5f1db23c7e73b84, ad50350dbfc945ec9a97f4bcf2c1ad9c; no voters:
I20260812 00:58:47.307549 30204 raft_consensus.cc:2781] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:58:47.307854 30204 raft_consensus.cc:482] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:58:47.307998 30204 raft_consensus.cc:3037] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:47.310492 30204 raft_consensus.cc:504] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773393c70b414223b943f6e7be481471" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34301 } } peers { permanent_uuid: "81206e5a82194f4cb5f1db23c7e73b84" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33785 } } peers { permanent_uuid: "ad50350dbfc945ec9a97f4bcf2c1ad9c" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 38725 } }
I20260812 00:58:47.311431 30204 leader_election.cc:288] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c [CANDIDATE]: Term 1 election: Requested vote from peers 773393c70b414223b943f6e7be481471 (127.21.201.67:34301), 81206e5a82194f4cb5f1db23c7e73b84 (127.21.201.66:33785)
I20260812 00:58:47.312063 30068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4156093e86a342b89f997da9f78eb3bf" candidate_uuid: "ad50350dbfc945ec9a97f4bcf2c1ad9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "773393c70b414223b943f6e7be481471"
I20260812 00:58:47.312120 29858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4156093e86a342b89f997da9f78eb3bf" candidate_uuid: "ad50350dbfc945ec9a97f4bcf2c1ad9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81206e5a82194f4cb5f1db23c7e73b84"
I20260812 00:58:47.312371 29858 raft_consensus.cc:3037] T 4156093e86a342b89f997da9f78eb3bf P 81206e5a82194f4cb5f1db23c7e73b84 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:47.312321 30068 raft_consensus.cc:3037] T 4156093e86a342b89f997da9f78eb3bf P 773393c70b414223b943f6e7be481471 [term 0 FOLLOWER]: Advancing to term 1
W20260812 00:58:47.315287 29928 tablet.cc:2369] T 4156093e86a342b89f997da9f78eb3bf P 81206e5a82194f4cb5f1db23c7e73b84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:58:47.315315 29858 raft_consensus.cc:2454] T 4156093e86a342b89f997da9f78eb3bf P 81206e5a82194f4cb5f1db23c7e73b84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad50350dbfc945ec9a97f4bcf2c1ad9c in term 1.
I20260812 00:58:47.315910 30068 raft_consensus.cc:2454] T 4156093e86a342b89f997da9f78eb3bf P 773393c70b414223b943f6e7be481471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad50350dbfc945ec9a97f4bcf2c1ad9c in term 1.
I20260812 00:58:47.316527 29527 leader_election.cc:302] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81206e5a82194f4cb5f1db23c7e73b84, ad50350dbfc945ec9a97f4bcf2c1ad9c; no voters:
I20260812 00:58:47.316915 30204 raft_consensus.cc:2781] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:58:47.317521 30204 raft_consensus.cc:686] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c [term 1 LEADER]: Becoming Leader. State: Replica: ad50350dbfc945ec9a97f4bcf2c1ad9c, State: Running, Role: LEADER
I20260812 00:58:47.318292 30204 consensus_queue.cc:237] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773393c70b414223b943f6e7be481471" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34301 } } peers { permanent_uuid: "81206e5a82194f4cb5f1db23c7e73b84" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33785 } } peers { permanent_uuid: "ad50350dbfc945ec9a97f4bcf2c1ad9c" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 38725 } }
I20260812 00:58:47.323648 29381 catalog_manager.cc:5495] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c reported cstate change: term changed from 0 to 1, leader changed from <none> to ad50350dbfc945ec9a97f4bcf2c1ad9c (127.21.201.65). New cstate: current_term: 1 leader_uuid: "ad50350dbfc945ec9a97f4bcf2c1ad9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "773393c70b414223b943f6e7be481471" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81206e5a82194f4cb5f1db23c7e73b84" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad50350dbfc945ec9a97f4bcf2c1ad9c" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 38725 } health_report { overall_health: HEALTHY } } }
I20260812 00:58:47.387280 30181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:47.381008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52712 (local address 127.21.201.65:38725)
0812 00:58:47.381208 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:47.381213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:47.381233 (+ 20us) server_negotiation.cc:407] Connection header received
0812 00:58:47.381461 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:47.381465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:47.381554 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:47.381699 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:58:47.382153 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.382894 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.384515 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.385048 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.385211 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:47.385844 (+ 633us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:58:47.386451 (+ 607us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:58:47.386571 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:47.386810 (+ 239us) server_negotiation.cc:299] Negotiation successful
0812 00:58:47.386954 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 00:58:47.396996 30068 raft_consensus.cc:1261] T 4156093e86a342b89f997da9f78eb3bf P 773393c70b414223b943f6e7be481471 [term 1 FOLLOWER]: Refusing update from remote peer ad50350dbfc945ec9a97f4bcf2c1ad9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:58:47.396996 29858 raft_consensus.cc:1261] T 4156093e86a342b89f997da9f78eb3bf P 81206e5a82194f4cb5f1db23c7e73b84 [term 1 FOLLOWER]: Refusing update from remote peer ad50350dbfc945ec9a97f4bcf2c1ad9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:58:47.398566 30204 consensus_queue.cc:1035] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "773393c70b414223b943f6e7be481471" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:58:47.399123 30212 consensus_queue.cc:1035] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "81206e5a82194f4cb5f1db23c7e73b84" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:58:47.415330 30217 mvcc.cc:205] Tried to move back new op lower bound from 7317488957004656640 to 7317488956699533312. Current Snapshot: MvccSnapshot[applied={T|T < 7317488957004656640}]
I20260812 00:58:47.433818 29379 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:58:47.434263 29379 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:58:47.439654 29379 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6c27c7589294cffac2dfb1b633c675f: Sending DeleteTablet for 3 replicas of tablet 4156093e86a342b89f997da9f78eb3bf
I20260812 00:58:47.441398 30042 tablet_service.cc:1514] Processing DeleteTablet for tablet 4156093e86a342b89f997da9f78eb3bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:58:47 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:55190
I20260812 00:58:47.441803 29595 tablet_service.cc:1514] Processing DeleteTablet for tablet 4156093e86a342b89f997da9f78eb3bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:58:47 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:34198
I20260812 00:58:47.442154 29822 tablet_service.cc:1514] Processing DeleteTablet for tablet 4156093e86a342b89f997da9f78eb3bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:58:47 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:35082
I20260812 00:58:47.445441 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 29494
I20260812 00:58:47.452397 30238 tablet_replica.cc:330] T 4156093e86a342b89f997da9f78eb3bf P 81206e5a82194f4cb5f1db23c7e73b84: stopping tablet replica
I20260812 00:58:47.453727 30238 raft_consensus.cc:2229] T 4156093e86a342b89f997da9f78eb3bf P 81206e5a82194f4cb5f1db23c7e73b84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:58:47.454878 30238 raft_consensus.cc:2258] T 4156093e86a342b89f997da9f78eb3bf P 81206e5a82194f4cb5f1db23c7e73b84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:58:47.455936 30237 tablet_replica.cc:330] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c: stopping tablet replica
I20260812 00:58:47.456918 30237 raft_consensus.cc:2229] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:58:47.457842 30237 raft_consensus.cc:2258] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:58:47.459656 30238 ts_tablet_manager.cc:1905] T 4156093e86a342b89f997da9f78eb3bf P 81206e5a82194f4cb5f1db23c7e73b84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:58:47.459893 30234 tablet_replica.cc:330] T 4156093e86a342b89f997da9f78eb3bf P 773393c70b414223b943f6e7be481471: stopping tablet replica
I20260812 00:58:47.460419 30234 raft_consensus.cc:2229] T 4156093e86a342b89f997da9f78eb3bf P 773393c70b414223b943f6e7be481471 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:58:47.460804 30237 ts_tablet_manager.cc:1905] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:58:47.461023 30234 raft_consensus.cc:2258] T 4156093e86a342b89f997da9f78eb3bf P 773393c70b414223b943f6e7be481471 [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 12 00:58:47 dist-test-slave-7f01 krb5kdc[29307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.65@KRBTEST.COM
I20260812 00:58:47.468094 30238 ts_tablet_manager.cc:1918] T 4156093e86a342b89f997da9f78eb3bf P 81206e5a82194f4cb5f1db23c7e73b84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:58:47.468501 30238 log.cc:1196] T 4156093e86a342b89f997da9f78eb3bf P 81206e5a82194f4cb5f1db23c7e73b84: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/4156093e86a342b89f997da9f78eb3bf
I20260812 00:58:47.469322 30238 ts_tablet_manager.cc:1939] T 4156093e86a342b89f997da9f78eb3bf P 81206e5a82194f4cb5f1db23c7e73b84: Deleting consensus metadata
I20260812 00:58:47.470575 30237 ts_tablet_manager.cc:1918] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:58:47.471611 29355 catalog_manager.cc:4841] TS 81206e5a82194f4cb5f1db23c7e73b84 (127.21.201.66:33785): tablet 4156093e86a342b89f997da9f78eb3bf (table test-table [id=3257c3814c88459980824bbc24008d84]) successfully deleted
I20260812 00:58:47.472360 30237 log.cc:1196] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-0/wal/wals/4156093e86a342b89f997da9f78eb3bf
I20260812 00:58:47.472620 30234 ts_tablet_manager.cc:1905] T 4156093e86a342b89f997da9f78eb3bf P 773393c70b414223b943f6e7be481471: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:58:47.473057 30237 ts_tablet_manager.cc:1939] T 4156093e86a342b89f997da9f78eb3bf P ad50350dbfc945ec9a97f4bcf2c1ad9c: Deleting consensus metadata
I20260812 00:58:47.475090 29355 catalog_manager.cc:4841] TS ad50350dbfc945ec9a97f4bcf2c1ad9c (127.21.201.65:38725): tablet 4156093e86a342b89f997da9f78eb3bf (table test-table [id=3257c3814c88459980824bbc24008d84]) successfully deleted
I20260812 00:58:47.478266 30181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:47.447177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52718 (local address 127.21.201.65:38725)
0812 00:58:47.447377 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:47.447384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:47.450288 (+ 2904us) server_negotiation.cc:407] Connection header received
0812 00:58:47.451498 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:47.451508 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:47.451638 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:47.451755 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:47.455766 (+ 4011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.457021 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.461483 (+ 4462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.462653 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.462843 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:47.469514 (+ 6671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:47.469532 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:47.469552 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:47.469603 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:47.474339 (+ 4736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:47.475419 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:47.475429 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:47.475432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:47.475642 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:47.476257 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:47.476265 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:47.476271 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:47.476769 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:47.477098 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:47.477514 (+ 416us) server_negotiation.cc:299] Negotiation successful
0812 00:58:47.477799 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":70}
I20260812 00:58:47.479102 29697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:47.480269 30234 ts_tablet_manager.cc:1918] T 4156093e86a342b89f997da9f78eb3bf P 773393c70b414223b943f6e7be481471: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:58:47.480499 30234 log.cc:1196] T 4156093e86a342b89f997da9f78eb3bf P 773393c70b414223b943f6e7be481471: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/4156093e86a342b89f997da9f78eb3bf
I20260812 00:58:47.481093 30234 ts_tablet_manager.cc:1939] T 4156093e86a342b89f997da9f78eb3bf P 773393c70b414223b943f6e7be481471: Deleting consensus metadata
I20260812 00:58:47.483211 29357 catalog_manager.cc:4841] TS 773393c70b414223b943f6e7be481471 (127.21.201.67:34301): tablet 4156093e86a342b89f997da9f78eb3bf (table test-table [id=3257c3814c88459980824bbc24008d84]) successfully deleted
I20260812 00:58:47.584337 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 29494
I20260812 00:58:47.617221 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 29714
Aug 12 00:58:47 dist-test-slave-7f01 krb5kdc[29307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.66@KRBTEST.COM
I20260812 00:58:47.634338 30179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:47.618051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41224 (local address 127.21.201.66:33785)
0812 00:58:47.618233 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:47.618238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:47.618257 (+ 19us) server_negotiation.cc:407] Connection header received
0812 00:58:47.618374 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:47.618378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:47.618442 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:47.618541 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:47.619994 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.621093 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.622641 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.623078 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.623193 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:47.627800 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:47.627830 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:47.627852 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:47.627901 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:47.631058 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:47.631889 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:47.631902 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:47.631908 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:47.632061 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:47.632813 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:47.632817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:47.632819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:47.633284 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:47.633528 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:47.633877 (+ 349us) server_negotiation.cc:299] Negotiation successful
0812 00:58:47.634066 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 00:58:47.635057 29920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:47.732059 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 29714
I20260812 00:58:47.765107 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 29941
Aug 12 00:58:47 dist-test-slave-7f01 krb5kdc[29307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.67@KRBTEST.COM
I20260812 00:58:47.784426 30177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:47.766235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57680 (local address 127.21.201.67:34301)
0812 00:58:47.766420 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:47.766430 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:47.766449 (+ 19us) server_negotiation.cc:407] Connection header received
0812 00:58:47.766512 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:47.766516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:47.766584 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:47.766684 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:47.768333 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.769327 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.770763 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:47.771320 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:47.771446 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:47.777395 (+ 5949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:47.777419 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:47.777436 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:47.777501 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:47.781052 (+ 3551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:47.781885 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:47.781892 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:47.781894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:47.782051 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:47.782716 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:47.782721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:47.782723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:47.783166 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:47.783411 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:47.783926 (+ 515us) server_negotiation.cc:299] Negotiation successful
0812 00:58:47.784138 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 00:58:47.785336 30138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:47.884368 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 29941
I20260812 00:58:47.909499 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 29328
I20260812 00:58:47.910986 29434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:48.033674 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 29328
2026-08-12T00:58:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (5176 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-12T00:58:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:58:48Z Disabled control of system clock
I20260812 00:58:48.127704 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:39039
--webserver_interface=127.21.201.126
--webserver_port=0
--builtin_ntp_servers=127.21.201.84:35489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:39039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 00:58:48.505155 30254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:48.505503 30254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:48.505653 30254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:48.513396 30254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:58:48.513550 30254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:48.513590 30254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:48.513685 30254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:58:48.513754 30254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:58:48.523643 30254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:35489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:39039
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:39039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:48.525269 30254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:48.527280 30254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:48.541285 30260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:48.541111 30264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:48.541129 30262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:48.542286 30254 server_base.cc:1034] running on GCE node
I20260812 00:58:48.543371 30254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:48.545481 30254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:48.546787 30254 hybrid_clock.cc:648] HybridClock initialized: now 1786496328546739 us; error 54 us; skew 500 ppm
I20260812 00:58:48.547247 30254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 00:58:48.550154 30254 webserver.cc:460] Webserver started at http://127.21.201.126:39297/ using document root <none> and password file <none>
I20260812 00:58:48.550961 30254 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:48.551107 30254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:48.551471 30254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:48.554877 30254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "bfc545ce9fe04b9daa2d993d6bc65da3"
format_stamp: "Formatted at 2026-08-12 00:58:48 on dist-test-slave-7f01"
server_key: "f0a7e28a90f90d36361e9939ccf9b91e"
server_key_iv: "8af6bc2b3fe15e4e9148d3943210a8f0"
server_key_version: "encryptionkey@0"
I20260812 00:58:48.556051 30254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "bfc545ce9fe04b9daa2d993d6bc65da3"
format_stamp: "Formatted at 2026-08-12 00:58:48 on dist-test-slave-7f01"
server_key: "f0a7e28a90f90d36361e9939ccf9b91e"
server_key_iv: "8af6bc2b3fe15e4e9148d3943210a8f0"
server_key_version: "encryptionkey@0"
I20260812 00:58:48.562356 30254 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 00:58:48.566341 30271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:48.568728 30254 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 00:58:48.569056 30254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "bfc545ce9fe04b9daa2d993d6bc65da3"
format_stamp: "Formatted at 2026-08-12 00:58:48 on dist-test-slave-7f01"
server_key: "f0a7e28a90f90d36361e9939ccf9b91e"
server_key_iv: "8af6bc2b3fe15e4e9148d3943210a8f0"
server_key_version: "encryptionkey@0"
I20260812 00:58:48.569423 30254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:48.577585 30254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:48.578655 30254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:48.579016 30254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:48.599591 30254 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:39039
I20260812 00:58:48.599628 30364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:39039 every 8 connection(s)
I20260812 00:58:48.601109 30254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 00:58:48.605198 30365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:48.607163 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 30254
I20260812 00:58:48.607412 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 00:58:48.608335 22309 external_mini_cluster.cc:1428] Setting key da8dc8a0bad3271c1c34b313e6d39334
I20260812 00:58:48.618245 30371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:48.610705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54080 (local address 127.21.201.126:39039)
0812 00:58:48.611900 (+ 1195us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:48.611912 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:48.611959 (+ 47us) server_negotiation.cc:407] Connection header received
0812 00:58:48.614110 (+ 2151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:48.614136 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:48.614605 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:48.615049 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:48.616387 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:48.616395 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:48.616407 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:58:48.616433 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:48.616498 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:58:48.616516 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:48.616760 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:48.616901 (+ 141us) server_negotiation.cc:299] Negotiation successful
0812 00:58:48.617272 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":410,"threads_started":1}
I20260812 00:58:48.622071 30365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfc545ce9fe04b9daa2d993d6bc65da3: Bootstrap starting.
I20260812 00:58:48.625437 30365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfc545ce9fe04b9daa2d993d6bc65da3: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:48.626570 30365 log.cc:824] T 00000000000000000000000000000000 P bfc545ce9fe04b9daa2d993d6bc65da3: Log is configured to *not* fsync() on all Append() calls
I20260812 00:58:48.629926 30365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfc545ce9fe04b9daa2d993d6bc65da3: No bootstrap required, opened a new log
I20260812 00:58:48.638057 30365 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfc545ce9fe04b9daa2d993d6bc65da3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfc545ce9fe04b9daa2d993d6bc65da3" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 39039 } }
I20260812 00:58:48.638499 30365 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfc545ce9fe04b9daa2d993d6bc65da3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:48.638638 30365 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfc545ce9fe04b9daa2d993d6bc65da3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfc545ce9fe04b9daa2d993d6bc65da3, State: Initialized, Role: FOLLOWER
I20260812 00:58:48.639398 30365 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfc545ce9fe04b9daa2d993d6bc65da3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfc545ce9fe04b9daa2d993d6bc65da3" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 39039 } }
I20260812 00:58:48.639683 30365 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfc545ce9fe04b9daa2d993d6bc65da3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:58:48.639911 30365 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfc545ce9fe04b9daa2d993d6bc65da3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:58:48.640103 30365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfc545ce9fe04b9daa2d993d6bc65da3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:48.642480 30365 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfc545ce9fe04b9daa2d993d6bc65da3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfc545ce9fe04b9daa2d993d6bc65da3" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 39039 } }
I20260812 00:58:48.643098 30365 leader_election.cc:302] T 00000000000000000000000000000000 P bfc545ce9fe04b9daa2d993d6bc65da3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfc545ce9fe04b9daa2d993d6bc65da3; no voters:
I20260812 00:58:48.643968 30365 leader_election.cc:288] T 00000000000000000000000000000000 P bfc545ce9fe04b9daa2d993d6bc65da3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:58:48.644205 30374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfc545ce9fe04b9daa2d993d6bc65da3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:58:48.645277 30374 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfc545ce9fe04b9daa2d993d6bc65da3 [term 1 LEADER]: Becoming Leader. State: Replica: bfc545ce9fe04b9daa2d993d6bc65da3, State: Running, Role: LEADER
I20260812 00:58:48.646021 30374 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfc545ce9fe04b9daa2d993d6bc65da3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfc545ce9fe04b9daa2d993d6bc65da3" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 39039 } }
I20260812 00:58:48.646582 30365 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfc545ce9fe04b9daa2d993d6bc65da3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:58:48.650408 30375 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfc545ce9fe04b9daa2d993d6bc65da3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfc545ce9fe04b9daa2d993d6bc65da3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfc545ce9fe04b9daa2d993d6bc65da3" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 39039 } } }
I20260812 00:58:48.650849 30375 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfc545ce9fe04b9daa2d993d6bc65da3 [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:48.650550 30376 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfc545ce9fe04b9daa2d993d6bc65da3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfc545ce9fe04b9daa2d993d6bc65da3. Latest consensus state: current_term: 1 leader_uuid: "bfc545ce9fe04b9daa2d993d6bc65da3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfc545ce9fe04b9daa2d993d6bc65da3" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 39039 } } }
I20260812 00:58:48.651088 30376 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfc545ce9fe04b9daa2d993d6bc65da3 [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:48.651979 30391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:58:48.659109 30391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:58:48.672067 30391 catalog_manager.cc:1347] Generated new cluster ID: 0d260e0078cb4e1f932617dd18dddfd9
I20260812 00:58:48.672241 30391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:58:48.709846 30391 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:58:48.710436 30391 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:58:48.718449 30391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfc545ce9fe04b9daa2d993d6bc65da3: Generated new TSK 0
I20260812 00:58:48.719632 30391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:58:48.725139 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:0
--local_ip_for_outbound_sockets=127.21.201.65
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:39039
--builtin_ntp_servers=127.21.201.84:35489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 00:58:49.116880 30413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:49.117308 30413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:49.117492 30413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:49.125531 30413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:49.125718 30413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:49.125917 30413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 00:58:49.136096 30413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:35489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:39039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:49.137552 30413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:49.139433 30413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:49.154352 30430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:49.154460 30425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:49.154457 30427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:49.155992 30413 server_base.cc:1034] running on GCE node
I20260812 00:58:49.156785 30413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:49.158479 30413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:49.159900 30413 hybrid_clock.cc:648] HybridClock initialized: now 1786496329159816 us; error 57 us; skew 500 ppm
I20260812 00:58:49.160426 30413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 00:58:49.163430 30413 webserver.cc:460] Webserver started at http://127.21.201.65:42411/ using document root <none> and password file <none>
I20260812 00:58:49.164305 30413 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:49.164454 30413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:49.164808 30413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:49.168110 30413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-0/data/instance:
uuid: "b3dfa76d835a46c5856b24ff53323d19"
format_stamp: "Formatted at 2026-08-12 00:58:49 on dist-test-slave-7f01"
server_key: "eef1a407e911d39e22c7528527efeb0a"
server_key_iv: "b7990f19a971a9ffcbee76ea8c22a1f4"
server_key_version: "encryptionkey@0"
I20260812 00:58:49.169035 30413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance:
uuid: "b3dfa76d835a46c5856b24ff53323d19"
format_stamp: "Formatted at 2026-08-12 00:58:49 on dist-test-slave-7f01"
server_key: "eef1a407e911d39e22c7528527efeb0a"
server_key_iv: "b7990f19a971a9ffcbee76ea8c22a1f4"
server_key_version: "encryptionkey@0"
I20260812 00:58:49.175007 30413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 00:58:49.179062 30439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:49.180896 30413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 00:58:49.181126 30413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "b3dfa76d835a46c5856b24ff53323d19"
format_stamp: "Formatted at 2026-08-12 00:58:49 on dist-test-slave-7f01"
server_key: "eef1a407e911d39e22c7528527efeb0a"
server_key_iv: "b7990f19a971a9ffcbee76ea8c22a1f4"
server_key_version: "encryptionkey@0"
I20260812 00:58:49.181392 30413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:49.188185 30413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:49.189146 30413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:49.189461 30413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:49.190626 30413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:58:49.192513 30413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:49.192653 30413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:49.192811 30413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:49.192911 30413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:49.233930 30413 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:35769
I20260812 00:58:49.234073 30637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:35769 every 8 connection(s)
I20260812 00:58:49.235452 30413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 00:58:49.244884 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 30413
I20260812 00:58:49.245128 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance
I20260812 00:58:49.245955 22309 external_mini_cluster.cc:1428] Setting key c4db8e2dc33bf9b408ed78af0dc5c120
I20260812 00:58:49.247141 30644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:49.238882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:33141 (local address 127.21.201.126:39039)
0812 00:58:49.239527 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:49.239536 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:49.241301 (+ 1765us) server_negotiation.cc:407] Connection header received
0812 00:58:49.243205 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:49.243211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:49.243305 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:49.243452 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:49.245382 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:49.245389 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:49.245392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:58:49.245418 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:49.245481 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:58:49.245496 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:49.245641 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:49.246725 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0812 00:58:49.246821 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":282,"threads_started":1}
I20260812 00:58:49.247777 30645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:49.239677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:39039 (local address 127.21.201.65:33141)
0812 00:58:49.241056 (+ 1379us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:49.241123 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:49.242836 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:49.243720 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:49.243730 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:49.243972 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:49.244867 (+ 895us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:58:49.244873 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:49.244942 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:58:49.245087 (+ 145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:49.245623 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:49.245636 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:49.245646 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 00:58:49.246006 (+ 360us) client_negotiation.cc:240] Negotiation successful
0812 00:58:49.246328 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"thread_start_us":420,"threads_started":1}
I20260812 00:58:49.249387 30641 heartbeater.cc:347] Connected to a master server at 127.21.201.126:39039
I20260812 00:58:49.249677 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:0
--local_ip_for_outbound_sockets=127.21.201.66
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:39039
--builtin_ntp_servers=127.21.201.84:35489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 00:58:49.249768 30641 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:49.250537 30641 heartbeater.cc:511] Master 127.21.201.126:39039 requested a full tablet report, sending...
I20260812 00:58:49.253747 30302 ts_manager.cc:194] Registered new tserver with Master: b3dfa76d835a46c5856b24ff53323d19 (127.21.201.65:35769)
W20260812 00:58:49.622735 30646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:49.623077 30646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:49.623178 30646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:49.631175 30646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:49.631317 30646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:49.631500 30646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 00:58:49.641383 30646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:35489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:39039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:49.642890 30646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:49.644848 30646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:49.659351 30652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:49.659415 30651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:49.659389 30655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:49.660607 30646 server_base.cc:1034] running on GCE node
I20260812 00:58:49.661479 30646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:49.662674 30646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:49.663942 30646 hybrid_clock.cc:648] HybridClock initialized: now 1786496329663886 us; error 64 us; skew 500 ppm
I20260812 00:58:49.664335 30646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 00:58:49.666963 30646 webserver.cc:460] Webserver started at http://127.21.201.66:33563/ using document root <none> and password file <none>
I20260812 00:58:49.667804 30646 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:49.667953 30646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:49.668323 30646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:49.671643 30646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-1/data/instance:
uuid: "0aac0353b569415fbbc65fa4358a704f"
format_stamp: "Formatted at 2026-08-12 00:58:49 on dist-test-slave-7f01"
server_key: "554cddb15349b2aed366852e7dd997ef"
server_key_iv: "19f666ab15315f240859ec53a60677d4"
server_key_version: "encryptionkey@0"
I20260812 00:58:49.672642 30646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance:
uuid: "0aac0353b569415fbbc65fa4358a704f"
format_stamp: "Formatted at 2026-08-12 00:58:49 on dist-test-slave-7f01"
server_key: "554cddb15349b2aed366852e7dd997ef"
server_key_iv: "19f666ab15315f240859ec53a60677d4"
server_key_version: "encryptionkey@0"
I20260812 00:58:49.679066 30646 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 00:58:49.683933 30667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:49.685621 30646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 00:58:49.685863 30646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "0aac0353b569415fbbc65fa4358a704f"
format_stamp: "Formatted at 2026-08-12 00:58:49 on dist-test-slave-7f01"
server_key: "554cddb15349b2aed366852e7dd997ef"
server_key_iv: "19f666ab15315f240859ec53a60677d4"
server_key_version: "encryptionkey@0"
I20260812 00:58:49.686146 30646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:49.692627 30646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:49.693650 30646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:49.693959 30646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:49.695044 30646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:58:49.696782 30646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:49.696921 30646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:49.697063 30646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:49.697155 30646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:49.734457 30646 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:43213
I20260812 00:58:49.734519 30851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:43213 every 8 connection(s)
I20260812 00:58:49.735996 30646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 00:58:49.741442 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 30646
I20260812 00:58:49.741674 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance
I20260812 00:58:49.742326 22309 external_mini_cluster.cc:1428] Setting key 7f66f79b79639884f94caf0457f3bdc5
I20260812 00:58:49.746104 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:0
--local_ip_for_outbound_sockets=127.21.201.67
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:39039
--builtin_ntp_servers=127.21.201.84:35489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 00:58:49.747124 30644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:49.739330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:53793 (local address 127.21.201.126:39039)
0812 00:58:49.739607 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:49.739615 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:49.741914 (+ 2299us) server_negotiation.cc:407] Connection header received
0812 00:58:49.744288 (+ 2374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:49.744293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:49.744362 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:49.744517 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:49.746260 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:49.746268 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:49.746273 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:58:49.746303 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:49.746358 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:58:49.746389 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:49.746505 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:49.746735 (+ 230us) server_negotiation.cc:299] Negotiation successful
0812 00:58:49.746806 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260812 00:58:49.748628 30855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:49.739906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:39039 (local address 127.21.201.66:53793)
0812 00:58:49.741666 (+ 1760us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:49.741734 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:49.743995 (+ 2261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:49.744797 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:49.744817 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:49.744955 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:49.745785 (+ 830us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:58:49.745789 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:49.745859 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:58:49.746010 (+ 151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:49.746498 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:49.746512 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:49.746523 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 00:58:49.746853 (+ 330us) client_negotiation.cc:240] Negotiation successful
0812 00:58:49.747161 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"thread_start_us":430,"threads_started":1}
I20260812 00:58:49.750864 30852 heartbeater.cc:347] Connected to a master server at 127.21.201.126:39039
I20260812 00:58:49.751260 30852 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:49.752132 30852 heartbeater.cc:511] Master 127.21.201.126:39039 requested a full tablet report, sending...
I20260812 00:58:49.754207 30302 ts_manager.cc:194] Registered new tserver with Master: 0aac0353b569415fbbc65fa4358a704f (127.21.201.66:43213)
W20260812 00:58:50.129117 30856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:50.129474 30856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:50.129630 30856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:50.137730 30856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:50.137861 30856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:50.138041 30856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 00:58:50.148716 30856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:35489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:39039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:50.150194 30856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:50.152221 30856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:50.167711 30867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:50.167884 30864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:50.167855 30865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:50.169906 30856 server_base.cc:1034] running on GCE node
I20260812 00:58:50.170670 30856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:50.171988 30856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:50.173393 30856 hybrid_clock.cc:648] HybridClock initialized: now 1786496330173301 us; error 74 us; skew 500 ppm
I20260812 00:58:50.174057 30856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 00:58:50.176942 30856 webserver.cc:460] Webserver started at http://127.21.201.67:38585/ using document root <none> and password file <none>
I20260812 00:58:50.177762 30856 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:50.177927 30856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:50.178300 30856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:50.181674 30856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-2/data/instance:
uuid: "ec064b2c26ef48c98e84f0331487abe3"
format_stamp: "Formatted at 2026-08-12 00:58:50 on dist-test-slave-7f01"
server_key: "8d5221fd6e8f67e4fc35bc219f35ccbc"
server_key_iv: "6ffd8e2169eb871b3a8ecd0cc46b7a8d"
server_key_version: "encryptionkey@0"
I20260812 00:58:50.182610 30856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance:
uuid: "ec064b2c26ef48c98e84f0331487abe3"
format_stamp: "Formatted at 2026-08-12 00:58:50 on dist-test-slave-7f01"
server_key: "8d5221fd6e8f67e4fc35bc219f35ccbc"
server_key_iv: "6ffd8e2169eb871b3a8ecd0cc46b7a8d"
server_key_version: "encryptionkey@0"
I20260812 00:58:50.189172 30856 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 00:58:50.193425 30882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:50.195220 30856 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 00:58:50.195544 30856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "ec064b2c26ef48c98e84f0331487abe3"
format_stamp: "Formatted at 2026-08-12 00:58:50 on dist-test-slave-7f01"
server_key: "8d5221fd6e8f67e4fc35bc219f35ccbc"
server_key_iv: "6ffd8e2169eb871b3a8ecd0cc46b7a8d"
server_key_version: "encryptionkey@0"
I20260812 00:58:50.195883 30856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:50.202371 30856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:50.203318 30856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:50.203634 30856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:50.205225 30856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:58:50.207556 30856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:50.207736 30856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:50.208019 30856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:50.208123 30856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:50.249480 30856 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:39267
I20260812 00:58:50.249606 31066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:39267 every 8 connection(s)
I20260812 00:58:50.251042 30856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 00:58:50.256803 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 30856
I20260812 00:58:50.257000 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance
I20260812 00:58:50.257612 22309 external_mini_cluster.cc:1428] Setting key a7780bd744a54dced61f960bb51fe696
I20260812 00:58:50.257861 30641 heartbeater.cc:503] Master 127.21.201.126:39039 was elected leader, sending a full tablet report...
I20260812 00:58:50.262575 31070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:50.254323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:41775 (local address 127.21.201.126:39039)
0812 00:58:50.254964 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:50.254969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:50.256878 (+ 1909us) server_negotiation.cc:407] Connection header received
0812 00:58:50.259915 (+ 3037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:50.259920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:50.260001 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:50.260136 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:50.261692 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:50.261698 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:50.261702 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:58:50.261721 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:50.261776 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:58:50.261789 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:50.261918 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:50.262129 (+ 211us) server_negotiation.cc:299] Negotiation successful
0812 00:58:50.262204 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":331,"threads_started":1}
I20260812 00:58:50.263521 31073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:50.255265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:39039 (local address 127.21.201.67:41775)
0812 00:58:50.256644 (+ 1379us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:50.256698 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:50.259557 (+ 2859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:50.260423 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:50.260432 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:50.260536 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:50.261344 (+ 808us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:58:50.261350 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:50.261410 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:58:50.261523 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:50.261894 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:50.261905 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:50.261913 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 00:58:50.262222 (+ 309us) client_negotiation.cc:240] Negotiation successful
0812 00:58:50.262495 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":463,"threads_started":1}
I20260812 00:58:50.265126 31067 heartbeater.cc:347] Connected to a master server at 127.21.201.126:39039
I20260812 00:58:50.265451 31067 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:50.266088 31067 heartbeater.cc:511] Master 127.21.201.126:39039 requested a full tablet report, sending...
I20260812 00:58:50.267612 30300 ts_manager.cc:194] Registered new tserver with Master: ec064b2c26ef48c98e84f0331487abe3 (127.21.201.67:39267)
I20260812 00:58:50.276062 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:58:50.285554 31070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:50.282169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54090 (local address 127.21.201.126:39039)
0812 00:58:50.282430 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:50.282441 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:50.282923 (+ 482us) server_negotiation.cc:407] Connection header received
0812 00:58:50.283033 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:50.283038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:50.283155 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:50.283282 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:50.284617 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:50.284624 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:50.284627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:58:50.284648 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:50.284694 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:58:50.284708 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:50.284838 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:50.285067 (+ 229us) server_negotiation.cc:299] Negotiation successful
0812 00:58:50.285308 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 00:58:50.291486 30300 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:58:50.300733 30300 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:58:50.325281 31095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:50.317151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:43213 (local address 127.0.0.1:45972)
0812 00:58:50.319571 (+ 2420us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:50.319589 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:50.319677 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:50.322100 (+ 2423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:50.322105 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:50.322135 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:50.322800 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:58:50.322805 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:50.322842 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:58:50.322954 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:50.324275 (+ 1321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:50.324286 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:50.324293 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 00:58:50.324551 (+ 258us) client_negotiation.cc:240] Negotiation successful
0812 00:58:50.324640 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2286,"thread_start_us":2595,"threads_started":1}
I20260812 00:58:50.325366 31097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:50.316854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:35769 (local address 127.0.0.1:38850)
0812 00:58:50.318714 (+ 1860us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:50.318770 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:50.318969 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:50.321542 (+ 2573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:50.321550 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:50.321674 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:50.322899 (+ 1225us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:58:50.322903 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:50.322916 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:58:50.322954 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:50.324345 (+ 1391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:50.324350 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:50.324355 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:58:50.324521 (+ 166us) client_negotiation.cc:240] Negotiation successful
0812 00:58:50.324673 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1500,"mutex_wait_us":32,"thread_start_us":2402,"threads_started":1}
I20260812 00:58:50.325989 31100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:50.317909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45972 (local address 127.21.201.66:43213)
0812 00:58:50.321472 (+ 3563us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:50.321483 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:50.321514 (+ 31us) server_negotiation.cc:407] Connection header received
0812 00:58:50.321585 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:50.321593 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:50.321873 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:50.322617 (+ 744us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:50.324050 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:50.324058 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:50.324080 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:58:50.324095 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:50.324141 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:58:50.324159 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:50.324324 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:50.325558 (+ 1234us) server_negotiation.cc:299] Negotiation successful
0812 00:58:50.325693 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3369,"thread_start_us":3103,"threads_started":1}
I20260812 00:58:50.325325 31099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:50.317400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:39267 (local address 127.0.0.1:40780)
0812 00:58:50.321876 (+ 4476us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:50.321897 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:50.322199 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:50.323145 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:50.323150 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:50.323172 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:50.323862 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:58:50.323865 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:50.323882 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:58:50.323927 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:50.324524 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:50.324528 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:50.324534 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 00:58:50.324671 (+ 137us) client_negotiation.cc:240] Negotiation successful
0812 00:58:50.324719 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4314,"thread_start_us":4158,"threads_started":1}
I20260812 00:58:50.326434 31105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:50.318344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38850 (local address 127.21.201.65:35769)
0812 00:58:50.320537 (+ 2193us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:50.320550 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:50.320604 (+ 54us) server_negotiation.cc:407] Connection header received
0812 00:58:50.320721 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:50.320732 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:50.321246 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:50.322756 (+ 1510us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:50.324078 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:50.324088 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:50.324100 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:58:50.324119 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:50.324176 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:58:50.324198 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:50.325834 (+ 1636us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:50.325902 (+ 68us) server_negotiation.cc:299] Negotiation successful
0812 00:58:50.326016 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1815,"thread_start_us":1635,"threads_started":1}
I20260812 00:58:50.327363 31103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:50.318175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40780 (local address 127.21.201.67:39267)
0812 00:58:50.320810 (+ 2635us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:50.320828 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:50.322459 (+ 1631us) server_negotiation.cc:407] Connection header received
0812 00:58:50.322556 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:50.322563 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:50.322920 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:50.323090 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:50.324165 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:50.324172 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:50.324183 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:58:50.324208 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:50.324260 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:58:50.324284 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:50.324465 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:50.325738 (+ 1273us) server_negotiation.cc:299] Negotiation successful
0812 00:58:50.325856 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2438,"thread_start_us":2159,"threads_started":1}
I20260812 00:58:50.330665 30964 tablet_service.cc:1467] Processing CreateTablet for tablet 3901587072684214ac47f7ae16d7623e (DEFAULT_TABLE table=test-table [id=582c216a7b774861aacb4df44db57bce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:50.330461 30742 tablet_service.cc:1467] Processing CreateTablet for tablet 3901587072684214ac47f7ae16d7623e (DEFAULT_TABLE table=test-table [id=582c216a7b774861aacb4df44db57bce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:50.330461 30519 tablet_service.cc:1467] Processing CreateTablet for tablet 3901587072684214ac47f7ae16d7623e (DEFAULT_TABLE table=test-table [id=582c216a7b774861aacb4df44db57bce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:58:50.332048 30964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3901587072684214ac47f7ae16d7623e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:50.332059 30742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3901587072684214ac47f7ae16d7623e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:50.332059 30519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3901587072684214ac47f7ae16d7623e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:50.349707 31106 tablet_bootstrap.cc:492] T 3901587072684214ac47f7ae16d7623e P ec064b2c26ef48c98e84f0331487abe3: Bootstrap starting.
I20260812 00:58:50.352752 31107 tablet_bootstrap.cc:492] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f: Bootstrap starting.
I20260812 00:58:50.353677 31108 tablet_bootstrap.cc:492] T 3901587072684214ac47f7ae16d7623e P b3dfa76d835a46c5856b24ff53323d19: Bootstrap starting.
I20260812 00:58:50.355309 31106 tablet_bootstrap.cc:654] T 3901587072684214ac47f7ae16d7623e P ec064b2c26ef48c98e84f0331487abe3: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:50.356887 31106 log.cc:824] T 3901587072684214ac47f7ae16d7623e P ec064b2c26ef48c98e84f0331487abe3: Log is configured to *not* fsync() on all Append() calls
I20260812 00:58:50.357365 31107 tablet_bootstrap.cc:654] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:50.357549 31108 tablet_bootstrap.cc:654] T 3901587072684214ac47f7ae16d7623e P b3dfa76d835a46c5856b24ff53323d19: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:50.358738 31107 log.cc:824] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f: Log is configured to *not* fsync() on all Append() calls
I20260812 00:58:50.358763 31108 log.cc:824] T 3901587072684214ac47f7ae16d7623e P b3dfa76d835a46c5856b24ff53323d19: Log is configured to *not* fsync() on all Append() calls
I20260812 00:58:50.360885 31106 tablet_bootstrap.cc:492] T 3901587072684214ac47f7ae16d7623e P ec064b2c26ef48c98e84f0331487abe3: No bootstrap required, opened a new log
I20260812 00:58:50.361294 31106 ts_tablet_manager.cc:1397] T 3901587072684214ac47f7ae16d7623e P ec064b2c26ef48c98e84f0331487abe3: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20260812 00:58:50.362041 31108 tablet_bootstrap.cc:492] T 3901587072684214ac47f7ae16d7623e P b3dfa76d835a46c5856b24ff53323d19: No bootstrap required, opened a new log
I20260812 00:58:50.362154 31107 tablet_bootstrap.cc:492] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f: No bootstrap required, opened a new log
I20260812 00:58:50.362327 31108 ts_tablet_manager.cc:1397] T 3901587072684214ac47f7ae16d7623e P b3dfa76d835a46c5856b24ff53323d19: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260812 00:58:50.362423 31107 ts_tablet_manager.cc:1397] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260812 00:58:50.370244 31108 raft_consensus.cc:348] T 3901587072684214ac47f7ae16d7623e P b3dfa76d835a46c5856b24ff53323d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3dfa76d835a46c5856b24ff53323d19" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35769 } } peers { permanent_uuid: "0aac0353b569415fbbc65fa4358a704f" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 43213 } } peers { permanent_uuid: "ec064b2c26ef48c98e84f0331487abe3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 39267 } }
I20260812 00:58:50.370729 31108 raft_consensus.cc:374] T 3901587072684214ac47f7ae16d7623e P b3dfa76d835a46c5856b24ff53323d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:50.370853 31108 raft_consensus.cc:729] T 3901587072684214ac47f7ae16d7623e P b3dfa76d835a46c5856b24ff53323d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3dfa76d835a46c5856b24ff53323d19, State: Initialized, Role: FOLLOWER
I20260812 00:58:50.371582 31108 consensus_queue.cc:260] T 3901587072684214ac47f7ae16d7623e P b3dfa76d835a46c5856b24ff53323d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3dfa76d835a46c5856b24ff53323d19" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35769 } } peers { permanent_uuid: "0aac0353b569415fbbc65fa4358a704f" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 43213 } } peers { permanent_uuid: "ec064b2c26ef48c98e84f0331487abe3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 39267 } }
I20260812 00:58:50.373155 31106 raft_consensus.cc:348] T 3901587072684214ac47f7ae16d7623e P ec064b2c26ef48c98e84f0331487abe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3dfa76d835a46c5856b24ff53323d19" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35769 } } peers { permanent_uuid: "0aac0353b569415fbbc65fa4358a704f" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 43213 } } peers { permanent_uuid: "ec064b2c26ef48c98e84f0331487abe3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 39267 } }
I20260812 00:58:50.373782 31106 raft_consensus.cc:374] T 3901587072684214ac47f7ae16d7623e P ec064b2c26ef48c98e84f0331487abe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:50.373936 31106 raft_consensus.cc:729] T 3901587072684214ac47f7ae16d7623e P ec064b2c26ef48c98e84f0331487abe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec064b2c26ef48c98e84f0331487abe3, State: Initialized, Role: FOLLOWER
I20260812 00:58:50.373718 31107 raft_consensus.cc:348] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3dfa76d835a46c5856b24ff53323d19" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35769 } } peers { permanent_uuid: "0aac0353b569415fbbc65fa4358a704f" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 43213 } } peers { permanent_uuid: "ec064b2c26ef48c98e84f0331487abe3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 39267 } }
I20260812 00:58:50.374230 31107 raft_consensus.cc:374] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:50.374372 31107 raft_consensus.cc:729] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aac0353b569415fbbc65fa4358a704f, State: Initialized, Role: FOLLOWER
I20260812 00:58:50.374578 31108 ts_tablet_manager.cc:1428] T 3901587072684214ac47f7ae16d7623e P b3dfa76d835a46c5856b24ff53323d19: Time spent starting tablet: real 0.012s user 0.005s sys 0.008s
I20260812 00:58:50.375108 31107 consensus_queue.cc:260] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3dfa76d835a46c5856b24ff53323d19" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35769 } } peers { permanent_uuid: "0aac0353b569415fbbc65fa4358a704f" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 43213 } } peers { permanent_uuid: "ec064b2c26ef48c98e84f0331487abe3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 39267 } }
I20260812 00:58:50.375128 31106 consensus_queue.cc:260] T 3901587072684214ac47f7ae16d7623e P ec064b2c26ef48c98e84f0331487abe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3dfa76d835a46c5856b24ff53323d19" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35769 } } peers { permanent_uuid: "0aac0353b569415fbbc65fa4358a704f" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 43213 } } peers { permanent_uuid: "ec064b2c26ef48c98e84f0331487abe3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 39267 } }
I20260812 00:58:50.377449 31067 heartbeater.cc:503] Master 127.21.201.126:39039 was elected leader, sending a full tablet report...
I20260812 00:58:50.378383 31106 ts_tablet_manager.cc:1428] T 3901587072684214ac47f7ae16d7623e P ec064b2c26ef48c98e84f0331487abe3: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260812 00:58:50.379556 30852 heartbeater.cc:503] Master 127.21.201.126:39039 was elected leader, sending a full tablet report...
I20260812 00:58:50.380326 31107 ts_tablet_manager.cc:1428] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
W20260812 00:58:50.488626 30853 tablet.cc:2369] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:58:50.488595 30643 tablet.cc:2369] T 3901587072684214ac47f7ae16d7623e P b3dfa76d835a46c5856b24ff53323d19: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:58:50.503072 31069 tablet.cc:2369] T 3901587072684214ac47f7ae16d7623e P ec064b2c26ef48c98e84f0331487abe3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:58:50.560025 31118 raft_consensus.cc:482] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:58:50.560366 31118 raft_consensus.cc:504] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3dfa76d835a46c5856b24ff53323d19" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35769 } } peers { permanent_uuid: "0aac0353b569415fbbc65fa4358a704f" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 43213 } } peers { permanent_uuid: "ec064b2c26ef48c98e84f0331487abe3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 39267 } }
I20260812 00:58:50.562131 31118 leader_election.cc:288] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3dfa76d835a46c5856b24ff53323d19 (127.21.201.65:35769), ec064b2c26ef48c98e84f0331487abe3 (127.21.201.67:39267)
I20260812 00:58:50.566288 31105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:50.562769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:56555 (local address 127.21.201.65:35769)
0812 00:58:50.563014 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:50.563023 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:50.563857 (+ 834us) server_negotiation.cc:407] Connection header received
0812 00:58:50.564080 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:50.564084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:50.564138 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:50.564247 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:50.565514 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:50.565537 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:50.565543 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:58:50.565562 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:50.565602 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:58:50.565619 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:50.565761 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:50.565911 (+ 150us) server_negotiation.cc:299] Negotiation successful
0812 00:58:50.566013 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 00:58:50.566332 31103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:50.562929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:42329 (local address 127.21.201.67:39267)
0812 00:58:50.563164 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:50.563173 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:50.563936 (+ 763us) server_negotiation.cc:407] Connection header received
0812 00:58:50.564104 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:50.564109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:50.564163 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:50.564276 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:50.565589 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:50.565594 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:50.565599 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:58:50.565617 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:50.565656 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:58:50.565675 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:50.565799 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:50.566029 (+ 230us) server_negotiation.cc:299] Negotiation successful
0812 00:58:50.566094 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 00:58:50.566582 31120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:50.562795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:39267 (local address 127.21.201.66:42329)
0812 00:58:50.563851 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:50.563866 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:50.563963 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:50.564296 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:50.564300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:50.564326 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:50.565302 (+ 976us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:58:50.565304 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:50.565317 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:58:50.565360 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:50.565832 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:50.565837 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:50.565843 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 00:58:50.566056 (+ 213us) client_negotiation.cc:240] Negotiation successful
0812 00:58:50.566112 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"thread_start_us":582,"threads_started":1}
I20260812 00:58:50.567101 31119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:50.562756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:35769 (local address 127.21.201.66:56555)
0812 00:58:50.563699 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:50.563717 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:50.563931 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:50.564278 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:50.564282 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:50.564307 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:50.565195 (+ 888us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:58:50.565198 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:50.565265 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:58:50.565325 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:50.565735 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:50.565740 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:50.565745 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:58:50.566579 (+ 834us) client_negotiation.cc:240] Negotiation successful
0812 00:58:50.566634 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"thread_start_us":538,"threads_started":1}
I20260812 00:58:50.567806 30991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3901587072684214ac47f7ae16d7623e" candidate_uuid: "0aac0353b569415fbbc65fa4358a704f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec064b2c26ef48c98e84f0331487abe3" is_pre_election: true
I20260812 00:58:50.568344 30991 raft_consensus.cc:2454] T 3901587072684214ac47f7ae16d7623e P ec064b2c26ef48c98e84f0331487abe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aac0353b569415fbbc65fa4358a704f in term 0.
I20260812 00:58:50.568743 30558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3901587072684214ac47f7ae16d7623e" candidate_uuid: "0aac0353b569415fbbc65fa4358a704f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3dfa76d835a46c5856b24ff53323d19" is_pre_election: true
I20260812 00:58:50.569084 30672 leader_election.cc:302] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aac0353b569415fbbc65fa4358a704f, ec064b2c26ef48c98e84f0331487abe3; no voters:
I20260812 00:58:50.569336 30558 raft_consensus.cc:2454] T 3901587072684214ac47f7ae16d7623e P b3dfa76d835a46c5856b24ff53323d19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aac0353b569415fbbc65fa4358a704f in term 0.
I20260812 00:58:50.569516 31118 raft_consensus.cc:2781] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:58:50.569679 31118 raft_consensus.cc:482] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:58:50.569850 31118 raft_consensus.cc:3037] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:50.572355 31118 raft_consensus.cc:504] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3dfa76d835a46c5856b24ff53323d19" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35769 } } peers { permanent_uuid: "0aac0353b569415fbbc65fa4358a704f" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 43213 } } peers { permanent_uuid: "ec064b2c26ef48c98e84f0331487abe3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 39267 } }
I20260812 00:58:50.573097 31118 leader_election.cc:288] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f [CANDIDATE]: Term 1 election: Requested vote from peers b3dfa76d835a46c5856b24ff53323d19 (127.21.201.65:35769), ec064b2c26ef48c98e84f0331487abe3 (127.21.201.67:39267)
I20260812 00:58:50.573719 30991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3901587072684214ac47f7ae16d7623e" candidate_uuid: "0aac0353b569415fbbc65fa4358a704f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec064b2c26ef48c98e84f0331487abe3"
I20260812 00:58:50.573722 30558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3901587072684214ac47f7ae16d7623e" candidate_uuid: "0aac0353b569415fbbc65fa4358a704f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3dfa76d835a46c5856b24ff53323d19"
I20260812 00:58:50.573959 30991 raft_consensus.cc:3037] T 3901587072684214ac47f7ae16d7623e P ec064b2c26ef48c98e84f0331487abe3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:50.573964 30558 raft_consensus.cc:3037] T 3901587072684214ac47f7ae16d7623e P b3dfa76d835a46c5856b24ff53323d19 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:50.576550 30558 raft_consensus.cc:2454] T 3901587072684214ac47f7ae16d7623e P b3dfa76d835a46c5856b24ff53323d19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aac0353b569415fbbc65fa4358a704f in term 1.
I20260812 00:58:50.576700 30991 raft_consensus.cc:2454] T 3901587072684214ac47f7ae16d7623e P ec064b2c26ef48c98e84f0331487abe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aac0353b569415fbbc65fa4358a704f in term 1.
I20260812 00:58:50.577171 30669 leader_election.cc:302] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aac0353b569415fbbc65fa4358a704f, b3dfa76d835a46c5856b24ff53323d19; no voters:
I20260812 00:58:50.577590 31118 raft_consensus.cc:2781] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:58:50.578119 31118 raft_consensus.cc:686] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f [term 1 LEADER]: Becoming Leader. State: Replica: 0aac0353b569415fbbc65fa4358a704f, State: Running, Role: LEADER
I20260812 00:58:50.578660 31118 consensus_queue.cc:237] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3dfa76d835a46c5856b24ff53323d19" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35769 } } peers { permanent_uuid: "0aac0353b569415fbbc65fa4358a704f" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 43213 } } peers { permanent_uuid: "ec064b2c26ef48c98e84f0331487abe3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 39267 } }
I20260812 00:58:50.584347 30300 catalog_manager.cc:5495] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0aac0353b569415fbbc65fa4358a704f (127.21.201.66). New cstate: current_term: 1 leader_uuid: "0aac0353b569415fbbc65fa4358a704f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3dfa76d835a46c5856b24ff53323d19" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0aac0353b569415fbbc65fa4358a704f" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 43213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec064b2c26ef48c98e84f0331487abe3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 39267 } health_report { overall_health: UNKNOWN } } }
I20260812 00:58:50.635587 31100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:50.633456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45976 (local address 127.21.201.66:43213)
0812 00:58:50.633666 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:50.633671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:50.633692 (+ 21us) server_negotiation.cc:407] Connection header received
0812 00:58:50.633799 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:50.633803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:50.633892 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:50.633993 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:50.634987 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:50.634991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:50.634994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:58:50.635017 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:50.635053 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:58:50.635066 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:50.635175 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:50.635297 (+ 122us) server_negotiation.cc:299] Negotiation successful
0812 00:58:50.635351 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260812 00:58:50.645303 30558 raft_consensus.cc:1261] T 3901587072684214ac47f7ae16d7623e P b3dfa76d835a46c5856b24ff53323d19 [term 1 FOLLOWER]: Refusing update from remote peer 0aac0353b569415fbbc65fa4358a704f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:58:50.645303 30991 raft_consensus.cc:1261] T 3901587072684214ac47f7ae16d7623e P ec064b2c26ef48c98e84f0331487abe3 [term 1 FOLLOWER]: Refusing update from remote peer 0aac0353b569415fbbc65fa4358a704f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:58:50.646493 31122 consensus_queue.cc:1035] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec064b2c26ef48c98e84f0331487abe3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 39267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:58:50.646878 31118 consensus_queue.cc:1035] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfa76d835a46c5856b24ff53323d19" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:58:50.677594 30300 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:58:50.677933 30300 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:58:50.681967 30300 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bfc545ce9fe04b9daa2d993d6bc65da3: Sending DeleteTablet for 3 replicas of tablet 3901587072684214ac47f7ae16d7623e
I20260812 00:58:50.683358 30742 tablet_service.cc:1514] Processing DeleteTablet for tablet 3901587072684214ac47f7ae16d7623e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:58:50 UTC) from {username='slave'} at 127.0.0.1:45972
I20260812 00:58:50.683351 30519 tablet_service.cc:1514] Processing DeleteTablet for tablet 3901587072684214ac47f7ae16d7623e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:58:50 UTC) from {username='slave'} at 127.0.0.1:38850
I20260812 00:58:50.683717 30964 tablet_service.cc:1514] Processing DeleteTablet for tablet 3901587072684214ac47f7ae16d7623e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:58:50 UTC) from {username='slave'} at 127.0.0.1:40780
I20260812 00:58:50.685962 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 30413
I20260812 00:58:50.695832 31147 tablet_replica.cc:330] T 3901587072684214ac47f7ae16d7623e P b3dfa76d835a46c5856b24ff53323d19: stopping tablet replica
I20260812 00:58:50.696393 31147 raft_consensus.cc:2229] T 3901587072684214ac47f7ae16d7623e P b3dfa76d835a46c5856b24ff53323d19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:58:50.696532 31151 tablet_replica.cc:330] T 3901587072684214ac47f7ae16d7623e P ec064b2c26ef48c98e84f0331487abe3: stopping tablet replica
I20260812 00:58:50.696936 31151 raft_consensus.cc:2229] T 3901587072684214ac47f7ae16d7623e P ec064b2c26ef48c98e84f0331487abe3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:58:50.696990 31147 raft_consensus.cc:2258] T 3901587072684214ac47f7ae16d7623e P b3dfa76d835a46c5856b24ff53323d19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:58:50.697431 31151 raft_consensus.cc:2258] T 3901587072684214ac47f7ae16d7623e P ec064b2c26ef48c98e84f0331487abe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:58:50.699013 31105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:50.688301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38852 (local address 127.21.201.65:35769)
0812 00:58:50.688606 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:50.688616 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:50.693694 (+ 5078us) server_negotiation.cc:407] Connection header received
0812 00:58:50.694645 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:50.694655 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:50.694995 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:50.695104 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:50.698199 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:50.698210 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:50.698216 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:58:50.698240 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:50.698284 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:58:50.698302 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:50.698478 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:50.698607 (+ 129us) server_negotiation.cc:299] Negotiation successful
0812 00:58:50.698678 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260812 00:58:50.700058 30630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:50.700346 31151 ts_tablet_manager.cc:1905] T 3901587072684214ac47f7ae16d7623e P ec064b2c26ef48c98e84f0331487abe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:58:50.703779 31148 tablet_replica.cc:330] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f: stopping tablet replica
I20260812 00:58:50.704309 31148 raft_consensus.cc:2229] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:58:50.705133 31148 raft_consensus.cc:2258] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:58:50.707957 31148 ts_tablet_manager.cc:1905] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:58:50.708844 31151 ts_tablet_manager.cc:1918] T 3901587072684214ac47f7ae16d7623e P ec064b2c26ef48c98e84f0331487abe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:58:50.709049 31151 log.cc:1196] T 3901587072684214ac47f7ae16d7623e P ec064b2c26ef48c98e84f0331487abe3: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/3901587072684214ac47f7ae16d7623e
I20260812 00:58:50.709587 31151 ts_tablet_manager.cc:1939] T 3901587072684214ac47f7ae16d7623e P ec064b2c26ef48c98e84f0331487abe3: Deleting consensus metadata
I20260812 00:58:50.711707 30277 catalog_manager.cc:4841] TS ec064b2c26ef48c98e84f0331487abe3 (127.21.201.67:39267): tablet 3901587072684214ac47f7ae16d7623e (table test-table [id=582c216a7b774861aacb4df44db57bce]) successfully deleted
I20260812 00:58:50.716789 31148 ts_tablet_manager.cc:1918] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:58:50.717034 31148 log.cc:1196] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/3901587072684214ac47f7ae16d7623e
I20260812 00:58:50.717665 31148 ts_tablet_manager.cc:1939] T 3901587072684214ac47f7ae16d7623e P 0aac0353b569415fbbc65fa4358a704f: Deleting consensus metadata
I20260812 00:58:50.719852 30276 catalog_manager.cc:4841] TS 0aac0353b569415fbbc65fa4358a704f (127.21.201.66:43213): tablet 3901587072684214ac47f7ae16d7623e (table test-table [id=582c216a7b774861aacb4df44db57bce]) successfully deleted
I20260812 00:58:50.700304 31147 ts_tablet_manager.cc:1905] T 3901587072684214ac47f7ae16d7623e P b3dfa76d835a46c5856b24ff53323d19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:58:50.809659 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 30413
I20260812 00:58:50.842676 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 30646
W20260812 00:58:50.842787 30272 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.201.65:35769 (error 108)
I20260812 00:58:50.845896 31160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:50.844617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:35769 (local address 127.0.0.1:38868)
0812 00:58:50.845465 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:50.845628 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:35769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":249,"threads_started":1}
I20260812 00:58:50.846589 31100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:50.843574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45990 (local address 127.21.201.66:43213)
0812 00:58:50.843939 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:50.843947 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:50.844006 (+ 59us) server_negotiation.cc:407] Connection header received
0812 00:58:50.844218 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:50.844222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:50.844281 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:50.844404 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:50.845397 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:50.845402 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:50.845408 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:58:50.845426 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:50.845467 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:58:50.845485 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:50.845623 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:50.845806 (+ 183us) server_negotiation.cc:299] Negotiation successful
0812 00:58:50.845872 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260812 00:58:50.846468 30272 catalog_manager.cc:4584] TS b3dfa76d835a46c5856b24ff53323d19 (127.21.201.65:35769): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3901587072684214ac47f7ae16d7623e: Network error: Client connection negotiation failed: client connection to 127.21.201.65:35769: connect: Connection refused (error 111)
I20260812 00:58:50.847548 30844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:50.900866 31160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:50.900366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:35769 (local address 127.0.0.1:38878)
0812 00:58:50.900617 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:50.900708 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:35769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260812 00:58:50.901999 31160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:50.901612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:35769 (local address 127.0.0.1:38880)
0812 00:58:50.901771 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:50.901852 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:35769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260812 00:58:50.962646 31160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:50.962057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:35769 (local address 127.0.0.1:38882)
0812 00:58:50.962290 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:50.962448 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:35769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260812 00:58:50.964600 31160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:50.963346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:35769 (local address 127.0.0.1:38886)
0812 00:58:50.964201 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:50.964383 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:35769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260812 00:58:50.965694 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 30646
I20260812 00:58:51.005089 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 30856
I20260812 00:58:51.008738 31103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:51.006028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40794 (local address 127.21.201.67:39267)
0812 00:58:51.006221 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:51.006226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:51.006246 (+ 20us) server_negotiation.cc:407] Connection header received
0812 00:58:51.006462 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:51.006467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:51.006573 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:51.006668 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:51.007681 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:51.007686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:51.007690 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:58:51.007706 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:51.007810 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:58:51.007834 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:51.007956 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:51.008259 (+ 303us) server_negotiation.cc:299] Negotiation successful
0812 00:58:51.008354 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 00:58:51.009675 31060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:51.045426 31160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:51.044871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:35769 (local address 127.0.0.1:38896)
0812 00:58:51.045169 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:51.045260 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:35769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260812 00:58:51.047259 31160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:51.046712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:35769 (local address 127.0.0.1:38910)
0812 00:58:51.046974 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:51.047089 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:35769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260812 00:58:51.105360 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 30856
I20260812 00:58:51.130470 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 30254
I20260812 00:58:51.131920 30359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:51.245980 31160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:51.245534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:35769 (local address 127.0.0.1:38912)
0812 00:58:51.245700 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:51.245822 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:35769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260812 00:58:51.246039 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 30254
2026-08-12T00:58:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3210 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260812 00:58:51.298097 22309 mini_oidc.cc:152] Starting JWKS server
I20260812 00:58:51.301345 22309 webserver.cc:460] Webserver started at https://127.0.0.1:36439/ using document root <none> and password file <none>
I20260812 00:58:51.301712 22309 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 00:58:51.302444 22309 webserver.cc:460] Webserver started at http://127.0.0.1:34899/ using document root <none> and password file <none>
2026-08-12T00:58:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:58:51Z Disabled control of system clock
I20260812 00:58:51.341858 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:35863
--webserver_interface=127.21.201.126
--webserver_port=0
--builtin_ntp_servers=127.21.201.84:42357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:35863
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36439/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786496278741276-22309-0/testchainca.cert with env {}
W20260812 00:58:51.725093 31176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:51.725523 31176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:51.725709 31176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:51.734037 31176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:58:51.734217 31176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:51.734323 31176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:51.734450 31176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:58:51.734535 31176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 00:58:51.734632 31176 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 00:58:51.734715 31176 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36439/jwks.json
I20260812 00:58:51.745602 31176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:42357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:35863
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:35863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36439/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786496278741276-22309-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:51.747179 31176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:51.748679 31176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:51.761651 31182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:51.761616 31183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:51.761703 31186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:51.762689 31176 server_base.cc:1034] running on GCE node
I20260812 00:58:51.763563 31176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:51.765147 31176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:51.766440 31176 hybrid_clock.cc:648] HybridClock initialized: now 1786496331766391 us; error 44 us; skew 500 ppm
I20260812 00:58:51.766857 31176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 00:58:51.775308 31176 webserver.cc:460] Webserver started at http://127.21.201.126:45535/ using document root <none> and password file <none>
I20260812 00:58:51.776266 31176 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:51.776445 31176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:51.776801 31176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:51.780126 31176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "5926475bcfbb4097b3bd39b9093ca35a"
format_stamp: "Formatted at 2026-08-12 00:58:51 on dist-test-slave-7f01"
server_key: "a78f4f9ce52f38a7ceeb03d80888157e"
server_key_iv: "e907ac8f847f3e23ad1316cad120a126"
server_key_version: "encryptionkey@0"
I20260812 00:58:51.781040 31176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "5926475bcfbb4097b3bd39b9093ca35a"
format_stamp: "Formatted at 2026-08-12 00:58:51 on dist-test-slave-7f01"
server_key: "a78f4f9ce52f38a7ceeb03d80888157e"
server_key_iv: "e907ac8f847f3e23ad1316cad120a126"
server_key_version: "encryptionkey@0"
I20260812 00:58:51.786887 31176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 00:58:51.790944 31196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:51.792762 31176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 00:58:51.793111 31176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "5926475bcfbb4097b3bd39b9093ca35a"
format_stamp: "Formatted at 2026-08-12 00:58:51 on dist-test-slave-7f01"
server_key: "a78f4f9ce52f38a7ceeb03d80888157e"
server_key_iv: "e907ac8f847f3e23ad1316cad120a126"
server_key_version: "encryptionkey@0"
I20260812 00:58:51.793392 31176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:51.843672 31176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:51.844723 31176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:51.845088 31176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:51.868727 31176 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:35863
I20260812 00:58:51.868752 31293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:35863 every 8 connection(s)
I20260812 00:58:51.870689 31176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 00:58:51.873085 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 31176
I20260812 00:58:51.873392 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 00:58:51.874466 22309 external_mini_cluster.cc:1428] Setting key 8da565b6cf05128de4c129f222a23f54
I20260812 00:58:51.878118 31294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:51.896651 31294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5926475bcfbb4097b3bd39b9093ca35a: Bootstrap starting.
I20260812 00:58:51.899447 31298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:51.877464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53070 (local address 127.21.201.126:35863)
0812 00:58:51.885852 (+ 8388us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:51.885866 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:51.885921 (+ 55us) server_negotiation.cc:407] Connection header received
0812 00:58:51.889548 (+ 3627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:51.889589 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:51.890443 (+ 854us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:51.890932 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:51.891566 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:51.892975 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:51.894371 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:51.895098 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:51.895314 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:51.896702 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:51.896711 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:51.896725 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:58:51.897358 (+ 633us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:51.897468 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:58:51.897494 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:51.897625 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:51.897882 (+ 257us) server_negotiation.cc:299] Negotiation successful
0812 00:58:51.898368 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7642,"thread_start_us":7338,"threads_started":1}
I20260812 00:58:51.901645 31294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5926475bcfbb4097b3bd39b9093ca35a: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:51.903095 31294 log.cc:824] T 00000000000000000000000000000000 P 5926475bcfbb4097b3bd39b9093ca35a: Log is configured to *not* fsync() on all Append() calls
I20260812 00:58:51.904742 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:58:51.906211 31294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5926475bcfbb4097b3bd39b9093ca35a: No bootstrap required, opened a new log
I20260812 00:58:51.913825 31294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5926475bcfbb4097b3bd39b9093ca35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5926475bcfbb4097b3bd39b9093ca35a" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 35863 } }
I20260812 00:58:51.914283 31294 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5926475bcfbb4097b3bd39b9093ca35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:51.914542 31294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5926475bcfbb4097b3bd39b9093ca35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5926475bcfbb4097b3bd39b9093ca35a, State: Initialized, Role: FOLLOWER
I20260812 00:58:51.915174 31294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5926475bcfbb4097b3bd39b9093ca35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5926475bcfbb4097b3bd39b9093ca35a" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 35863 } }
I20260812 00:58:51.915465 31294 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5926475bcfbb4097b3bd39b9093ca35a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:58:51.915634 31294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5926475bcfbb4097b3bd39b9093ca35a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:58:51.915853 31294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5926475bcfbb4097b3bd39b9093ca35a [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:51.918253 31294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5926475bcfbb4097b3bd39b9093ca35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5926475bcfbb4097b3bd39b9093ca35a" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 35863 } }
I20260812 00:58:51.918870 31294 leader_election.cc:302] T 00000000000000000000000000000000 P 5926475bcfbb4097b3bd39b9093ca35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5926475bcfbb4097b3bd39b9093ca35a; no voters:
I20260812 00:58:51.919807 31294 leader_election.cc:288] T 00000000000000000000000000000000 P 5926475bcfbb4097b3bd39b9093ca35a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:58:51.920001 31302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5926475bcfbb4097b3bd39b9093ca35a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:58:51.921097 31302 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5926475bcfbb4097b3bd39b9093ca35a [term 1 LEADER]: Becoming Leader. State: Replica: 5926475bcfbb4097b3bd39b9093ca35a, State: Running, Role: LEADER
I20260812 00:58:51.921738 31302 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5926475bcfbb4097b3bd39b9093ca35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5926475bcfbb4097b3bd39b9093ca35a" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 35863 } }
I20260812 00:58:51.922374 31294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5926475bcfbb4097b3bd39b9093ca35a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:58:51.925791 31303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5926475bcfbb4097b3bd39b9093ca35a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5926475bcfbb4097b3bd39b9093ca35a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5926475bcfbb4097b3bd39b9093ca35a" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 35863 } } }
I20260812 00:58:51.926024 31304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5926475bcfbb4097b3bd39b9093ca35a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5926475bcfbb4097b3bd39b9093ca35a. Latest consensus state: current_term: 1 leader_uuid: "5926475bcfbb4097b3bd39b9093ca35a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5926475bcfbb4097b3bd39b9093ca35a" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 35863 } } }
I20260812 00:58:51.926329 31303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5926475bcfbb4097b3bd39b9093ca35a [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:51.926329 31304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5926475bcfbb4097b3bd39b9093ca35a [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:51.928138 31310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:58:51.933403 31310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:58:51.943599 31310 catalog_manager.cc:1347] Generated new cluster ID: 15537246dd704862a119a2fe5a3b5fa9
I20260812 00:58:51.943765 31310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:58:51.957024 31310 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:58:51.959040 31310 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:58:51.967135 31310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5926475bcfbb4097b3bd39b9093ca35a: Generated new TSK 0
I20260812 00:58:51.968309 31310 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 00:58:52.020097 31338 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260812 00:58:52.020519 31338 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 00:58:52.021944 31298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:51.996428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53074 (local address 127.21.201.126:35863)
0812 00:58:51.996720 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:51.996729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:51.997610 (+ 881us) server_negotiation.cc:407] Connection header received
0812 00:58:51.998320 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:51.998331 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:51.998467 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:51.998790 (+ 323us) server_negotiation.cc:226] Negotiated authn=JWT
0812 00:58:51.999832 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:52.001032 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:52.003203 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:52.003732 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:52.003959 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:52.004617 (+ 658us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 00:58:52.020993 (+ 16376us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 00:58:52.021547 (+ 554us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53074: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":87}
W20260812 00:58:52.022104 31298 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53074: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260812 00:58:52.024317 31337 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:58:51.996427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:35863 (local address 127.0.0.1:53074)
0812 00:58:51.997385 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:51.997415 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:51.997692 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:51.998805 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:51.998811 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:51.998826 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0812 00:58:51.999485 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:51.999508 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:52.001362 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:52.001371 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:52.002990 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:52.003022 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:52.004041 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:52.004050 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:52.004211 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:52.004286 (+ 75us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 00:58:52.021437 (+ 17151us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 00:58:52.021870 (+ 433us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.201.126:35863: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":486,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.21.201.126:35863: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260812 00:58:52.026996 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 31176
I20260812 00:58:52.028625 31283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:52.133379 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 31176
2026-08-12T00:58:52Z chronyd exiting
I20260812 00:58:52.196650 22309 test_util.cc:175] -----------------------------------------------
I20260812 00:58:52.196859 22309 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786496278741276-22309-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (904 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260812 00:58:52.202231 22309 mini_oidc.cc:152] Starting JWKS server
I20260812 00:58:52.205010 22309 webserver.cc:460] Webserver started at https://127.0.0.1:44157/ using document root <none> and password file <none>
I20260812 00:58:52.205375 22309 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 00:58:52.206284 22309 webserver.cc:460] Webserver started at http://127.0.0.1:39895/ using document root <none> and password file <none>
2026-08-12T00:58:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:58:52Z Disabled control of system clock
I20260812 00:58:52.251039 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:34287
--webserver_interface=127.21.201.126
--webserver_port=0
--builtin_ntp_servers=127.21.201.84:34455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:34287
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44157/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786496278741276-22309-0/testchainca.cert with env {}
W20260812 00:58:52.650133 31351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:52.650487 31351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:52.650641 31351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:52.658457 31351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:58:52.658628 31351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:52.658739 31351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:52.658862 31351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:58:52.659087 31351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 00:58:52.659246 31351 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 00:58:52.659313 31351 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44157/jwks.json
I20260812 00:58:52.670145 31351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:34455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:34287
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:34287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44157/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786496278741276-22309-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:52.671700 31351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:52.673171 31351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:52.686878 31362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:52.686937 31359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:52.686987 31358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:52.692993 31351 server_base.cc:1034] running on GCE node
I20260812 00:58:52.694003 31351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:52.695688 31351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:52.697158 31351 hybrid_clock.cc:648] HybridClock initialized: now 1786496332697056 us; error 38 us; skew 500 ppm
I20260812 00:58:52.697933 31351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 00:58:52.701501 31351 webserver.cc:460] Webserver started at http://127.21.201.126:34163/ using document root <none> and password file <none>
I20260812 00:58:52.702348 31351 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:52.702517 31351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:52.702911 31351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:52.706367 31351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "3dd1ccbf7b1e456c95aa8abce4f74df6"
format_stamp: "Formatted at 2026-08-12 00:58:52 on dist-test-slave-7f01"
server_key: "b26e90213c7c036b9935d921cec41495"
server_key_iv: "e14a5ecaf2816968c8e1562494b9c93f"
server_key_version: "encryptionkey@0"
I20260812 00:58:52.707369 31351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "3dd1ccbf7b1e456c95aa8abce4f74df6"
format_stamp: "Formatted at 2026-08-12 00:58:52 on dist-test-slave-7f01"
server_key: "b26e90213c7c036b9935d921cec41495"
server_key_iv: "e14a5ecaf2816968c8e1562494b9c93f"
server_key_version: "encryptionkey@0"
I20260812 00:58:52.713588 31351 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260812 00:58:52.717499 31373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:52.719084 31351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 00:58:52.719408 31351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "3dd1ccbf7b1e456c95aa8abce4f74df6"
format_stamp: "Formatted at 2026-08-12 00:58:52 on dist-test-slave-7f01"
server_key: "b26e90213c7c036b9935d921cec41495"
server_key_iv: "e14a5ecaf2816968c8e1562494b9c93f"
server_key_version: "encryptionkey@0"
I20260812 00:58:52.719691 31351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:52.749871 31351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:52.750861 31351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:52.751217 31351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:52.773192 31351 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:34287
I20260812 00:58:52.773310 31459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:34287 every 8 connection(s)
I20260812 00:58:52.774713 31351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 00:58:52.778908 31460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:52.783955 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 31351
I20260812 00:58:52.784194 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 00:58:52.785100 22309 external_mini_cluster.cc:1428] Setting key 9844ba0b16562941b31ff30be4ee3ebf
I20260812 00:58:52.791865 31460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dd1ccbf7b1e456c95aa8abce4f74df6: Bootstrap starting.
I20260812 00:58:52.795943 31460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dd1ccbf7b1e456c95aa8abce4f74df6: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:52.798100 31460 log.cc:824] T 00000000000000000000000000000000 P 3dd1ccbf7b1e456c95aa8abce4f74df6: Log is configured to *not* fsync() on all Append() calls
I20260812 00:58:52.802533 31460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dd1ccbf7b1e456c95aa8abce4f74df6: No bootstrap required, opened a new log
I20260812 00:58:52.802769 31464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:52.787937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50996 (local address 127.21.201.126:34287)
0812 00:58:52.789175 (+ 1238us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:52.789187 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:52.789237 (+ 50us) server_negotiation.cc:407] Connection header received
0812 00:58:52.791175 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:52.791200 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:52.791708 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:52.792219 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:52.792882 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:52.794232 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:52.796202 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:52.796976 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:52.797133 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:52.799223 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:52.799237 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:52.799258 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:58:52.800255 (+ 997us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:52.800379 (+ 124us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:58:52.800415 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:52.800579 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:52.800864 (+ 285us) server_negotiation.cc:299] Negotiation successful
0812 00:58:52.801485 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":484,"threads_started":1}
I20260812 00:58:52.807888 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:58:52.812402 31460 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dd1ccbf7b1e456c95aa8abce4f74df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd1ccbf7b1e456c95aa8abce4f74df6" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 34287 } }
I20260812 00:58:52.812799 31460 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dd1ccbf7b1e456c95aa8abce4f74df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:52.812871 31460 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dd1ccbf7b1e456c95aa8abce4f74df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd1ccbf7b1e456c95aa8abce4f74df6, State: Initialized, Role: FOLLOWER
I20260812 00:58:52.813393 31460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dd1ccbf7b1e456c95aa8abce4f74df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd1ccbf7b1e456c95aa8abce4f74df6" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 34287 } }
I20260812 00:58:52.813596 31460 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dd1ccbf7b1e456c95aa8abce4f74df6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:58:52.813696 31460 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dd1ccbf7b1e456c95aa8abce4f74df6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:58:52.813827 31460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dd1ccbf7b1e456c95aa8abce4f74df6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:52.816196 31460 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dd1ccbf7b1e456c95aa8abce4f74df6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd1ccbf7b1e456c95aa8abce4f74df6" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 34287 } }
I20260812 00:58:52.816820 31460 leader_election.cc:302] T 00000000000000000000000000000000 P 3dd1ccbf7b1e456c95aa8abce4f74df6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dd1ccbf7b1e456c95aa8abce4f74df6; no voters:
I20260812 00:58:52.817440 31460 leader_election.cc:288] T 00000000000000000000000000000000 P 3dd1ccbf7b1e456c95aa8abce4f74df6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:58:52.817593 31466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dd1ccbf7b1e456c95aa8abce4f74df6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:58:52.818403 31466 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dd1ccbf7b1e456c95aa8abce4f74df6 [term 1 LEADER]: Becoming Leader. State: Replica: 3dd1ccbf7b1e456c95aa8abce4f74df6, State: Running, Role: LEADER
I20260812 00:58:52.819016 31466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dd1ccbf7b1e456c95aa8abce4f74df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd1ccbf7b1e456c95aa8abce4f74df6" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 34287 } }
I20260812 00:58:52.819947 31460 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dd1ccbf7b1e456c95aa8abce4f74df6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:58:52.822372 31469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dd1ccbf7b1e456c95aa8abce4f74df6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dd1ccbf7b1e456c95aa8abce4f74df6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd1ccbf7b1e456c95aa8abce4f74df6" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 34287 } } }
I20260812 00:58:52.822515 31470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dd1ccbf7b1e456c95aa8abce4f74df6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dd1ccbf7b1e456c95aa8abce4f74df6. Latest consensus state: current_term: 1 leader_uuid: "3dd1ccbf7b1e456c95aa8abce4f74df6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd1ccbf7b1e456c95aa8abce4f74df6" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 34287 } } }
I20260812 00:58:52.822865 31469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dd1ccbf7b1e456c95aa8abce4f74df6 [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:52.822932 31470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dd1ccbf7b1e456c95aa8abce4f74df6 [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:52.828861 31482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:58:52.833622 31482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:58:52.842796 31482 catalog_manager.cc:1347] Generated new cluster ID: e4572cfa3e5f415ebfbfa49f1b457008
I20260812 00:58:52.842960 31482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:58:52.856892 31482 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:58:52.859009 31482 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:58:52.869000 31482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3dd1ccbf7b1e456c95aa8abce4f74df6: Generated new TSK 0
I20260812 00:58:52.870185 31482 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 00:58:52.920559 31509 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260812 00:58:52.920768 31509 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 00:58:52.922390 31464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:52.892046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51002 (local address 127.21.201.126:34287)
0812 00:58:52.892370 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:52.892383 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:52.893015 (+ 632us) server_negotiation.cc:407] Connection header received
0812 00:58:52.893335 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:52.893341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:52.893429 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:52.893590 (+ 161us) server_negotiation.cc:226] Negotiated authn=JWT
0812 00:58:52.894245 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:52.895176 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:52.897369 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:52.897944 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:52.898120 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:52.898885 (+ 765us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 00:58:52.921354 (+ 22469us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 00:58:52.921815 (+ 461us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51002: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":100}
W20260812 00:58:52.922515 31508 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:58:52.891974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:34287 (local address 127.0.0.1:51002)
0812 00:58:52.892813 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:52.892844 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:52.893047 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:52.893642 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:52.893648 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:52.893662 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0812 00:58:52.894035 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:52.894048 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:52.895353 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:52.895359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:52.897106 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:52.897122 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:52.898193 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:52.898200 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:52.898427 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:52.898486 (+ 59us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 00:58:52.921697 (+ 23211us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 00:58:52.921981 (+ 284us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.201.126:34287: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":448,"threads_started":1}
W20260812 00:58:52.922581 31464 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51002: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260812 00:58:52.924078 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 31351
I20260812 00:58:52.925627 31449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:53.029825 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 31351
2026-08-12T00:58:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (900 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260812 00:58:53.103434 22309 mini_oidc.cc:152] Starting JWKS server
I20260812 00:58:53.105623 22309 webserver.cc:460] Webserver started at https://127.0.0.1:33441/ using document root <none> and password file <none>
I20260812 00:58:53.105927 22309 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 00:58:53.106658 22309 webserver.cc:460] Webserver started at http://127.0.0.1:35001/ using document root <none> and password file <none>
2026-08-12T00:58:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:58:53Z Disabled control of system clock
I20260812 00:58:53.145815 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:43173
--webserver_interface=127.21.201.126
--webserver_port=0
--builtin_ntp_servers=127.21.201.84:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:43173
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33441/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 00:58:53.519644 31518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:53.520110 31518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:53.520211 31518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:53.528486 31518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:58:53.528705 31518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:53.528859 31518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:53.528995 31518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:58:53.529064 31518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 00:58:53.529170 31518 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 00:58:53.529237 31518 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33441/jwks.json
I20260812 00:58:53.541098 31518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:43173
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:43173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33441/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:53.542742 31518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:53.544848 31518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:53.558301 31531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:53.558372 31538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:53.558406 31532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:53.559571 31518 server_base.cc:1034] running on GCE node
I20260812 00:58:53.560529 31518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:53.562230 31518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:53.563479 31518 hybrid_clock.cc:648] HybridClock initialized: now 1786496333563420 us; error 44 us; skew 500 ppm
I20260812 00:58:53.563987 31518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 00:58:53.567198 31518 webserver.cc:460] Webserver started at http://127.21.201.126:39649/ using document root <none> and password file <none>
I20260812 00:58:53.568338 31518 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:53.568461 31518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:53.568876 31518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:53.572858 31518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "8196ff5a28054c6b81c6b7ef310f14e6"
format_stamp: "Formatted at 2026-08-12 00:58:53 on dist-test-slave-7f01"
server_key: "573711ce2e7ab5c867324aef0fca19e7"
server_key_iv: "85478bc5540e373ebe4ffc7032891322"
server_key_version: "encryptionkey@0"
I20260812 00:58:53.574385 31518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "8196ff5a28054c6b81c6b7ef310f14e6"
format_stamp: "Formatted at 2026-08-12 00:58:53 on dist-test-slave-7f01"
server_key: "573711ce2e7ab5c867324aef0fca19e7"
server_key_iv: "85478bc5540e373ebe4ffc7032891322"
server_key_version: "encryptionkey@0"
I20260812 00:58:53.580988 31518 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 00:58:53.585439 31548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:53.587219 31518 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 00:58:53.587476 31518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "8196ff5a28054c6b81c6b7ef310f14e6"
format_stamp: "Formatted at 2026-08-12 00:58:53 on dist-test-slave-7f01"
server_key: "573711ce2e7ab5c867324aef0fca19e7"
server_key_iv: "85478bc5540e373ebe4ffc7032891322"
server_key_version: "encryptionkey@0"
I20260812 00:58:53.587721 31518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:53.617074 31518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:53.618074 31518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:53.618435 31518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:53.641819 31518 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:43173
I20260812 00:58:53.641834 31629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:43173 every 8 connection(s)
I20260812 00:58:53.643430 31518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 00:58:53.645298 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 31518
I20260812 00:58:53.645558 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 00:58:53.646797 22309 external_mini_cluster.cc:1428] Setting key 7d1d3be404509fe24d1860c525e033cd
I20260812 00:58:53.650148 31630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:53.662837 31630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8196ff5a28054c6b81c6b7ef310f14e6: Bootstrap starting.
I20260812 00:58:53.663419 31637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:53.649758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40996 (local address 127.21.201.126:43173)
0812 00:58:53.650924 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:53.650936 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:53.651000 (+ 64us) server_negotiation.cc:407] Connection header received
0812 00:58:53.652839 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:53.652863 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:53.653429 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:53.653881 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:53.654520 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:53.655993 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:53.657410 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:53.658122 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:53.658327 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:53.660073 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:53.660085 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:53.660114 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:58:53.661095 (+ 981us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:53.661319 (+ 224us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:58:53.661366 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:53.661566 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:53.661845 (+ 279us) server_negotiation.cc:299] Negotiation successful
0812 00:58:53.662242 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":450,"threads_started":1}
I20260812 00:58:53.666559 31630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8196ff5a28054c6b81c6b7ef310f14e6: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:53.667873 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:58:53.667989 31630 log.cc:824] T 00000000000000000000000000000000 P 8196ff5a28054c6b81c6b7ef310f14e6: Log is configured to *not* fsync() on all Append() calls
I20260812 00:58:53.671373 31630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8196ff5a28054c6b81c6b7ef310f14e6: No bootstrap required, opened a new log
I20260812 00:58:53.679512 31630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8196ff5a28054c6b81c6b7ef310f14e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8196ff5a28054c6b81c6b7ef310f14e6" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 43173 } }
I20260812 00:58:53.680075 31630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8196ff5a28054c6b81c6b7ef310f14e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:53.680210 31630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8196ff5a28054c6b81c6b7ef310f14e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8196ff5a28054c6b81c6b7ef310f14e6, State: Initialized, Role: FOLLOWER
I20260812 00:58:53.680907 31630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8196ff5a28054c6b81c6b7ef310f14e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8196ff5a28054c6b81c6b7ef310f14e6" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 43173 } }
I20260812 00:58:53.681174 31630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8196ff5a28054c6b81c6b7ef310f14e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:58:53.681339 31630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8196ff5a28054c6b81c6b7ef310f14e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:58:53.681538 31630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8196ff5a28054c6b81c6b7ef310f14e6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:53.684347 31630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8196ff5a28054c6b81c6b7ef310f14e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8196ff5a28054c6b81c6b7ef310f14e6" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 43173 } }
I20260812 00:58:53.684933 31630 leader_election.cc:302] T 00000000000000000000000000000000 P 8196ff5a28054c6b81c6b7ef310f14e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8196ff5a28054c6b81c6b7ef310f14e6; no voters:
I20260812 00:58:53.685601 31630 leader_election.cc:288] T 00000000000000000000000000000000 P 8196ff5a28054c6b81c6b7ef310f14e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:58:53.685811 31641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8196ff5a28054c6b81c6b7ef310f14e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:58:53.686636 31641 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8196ff5a28054c6b81c6b7ef310f14e6 [term 1 LEADER]: Becoming Leader. State: Replica: 8196ff5a28054c6b81c6b7ef310f14e6, State: Running, Role: LEADER
I20260812 00:58:53.687265 31641 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8196ff5a28054c6b81c6b7ef310f14e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8196ff5a28054c6b81c6b7ef310f14e6" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 43173 } }
I20260812 00:58:53.688087 31630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8196ff5a28054c6b81c6b7ef310f14e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:58:53.690737 31642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8196ff5a28054c6b81c6b7ef310f14e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8196ff5a28054c6b81c6b7ef310f14e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8196ff5a28054c6b81c6b7ef310f14e6" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 43173 } } }
I20260812 00:58:53.690824 31643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8196ff5a28054c6b81c6b7ef310f14e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8196ff5a28054c6b81c6b7ef310f14e6. Latest consensus state: current_term: 1 leader_uuid: "8196ff5a28054c6b81c6b7ef310f14e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8196ff5a28054c6b81c6b7ef310f14e6" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 43173 } } }
I20260812 00:58:53.691179 31642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8196ff5a28054c6b81c6b7ef310f14e6 [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:53.691265 31643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8196ff5a28054c6b81c6b7ef310f14e6 [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:53.697335 31652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:58:53.702775 31652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:58:53.711977 31652 catalog_manager.cc:1347] Generated new cluster ID: 30f4f36f830f4f21b662ae5817685ec4
I20260812 00:58:53.712123 31652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:58:53.745756 31652 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:58:53.747248 31652 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:58:53.758952 31652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8196ff5a28054c6b81c6b7ef310f14e6: Generated new TSK 0
I20260812 00:58:53.759999 31652 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:58:53.787045 31637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:53.776014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41000 (local address 127.21.201.126:43173)
0812 00:58:53.776292 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:53.776302 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:53.777006 (+ 704us) server_negotiation.cc:407] Connection header received
0812 00:58:53.777437 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:53.777443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:53.777535 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:53.777708 (+ 173us) server_negotiation.cc:226] Negotiated authn=JWT
0812 00:58:53.778482 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:53.779338 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:53.782111 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:53.782616 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:53.782737 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:53.783604 (+ 867us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 00:58:53.786309 (+ 2705us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33441/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 00:58:53.786680 (+ 371us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41000: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33441/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":83}
W20260812 00:58:53.787200 31637 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41000: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33441/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260812 00:58:53.789774 31682 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:58:53.775914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:43173 (local address 127.0.0.1:41000)
0812 00:58:53.776798 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:53.776837 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:53.777109 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:53.777769 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:53.777775 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:53.777791 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0812 00:58:53.778269 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:53.778284 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:53.779519 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:53.779525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:53.781799 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:53.781825 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:53.782883 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:53.782892 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:53.783148 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:53.783213 (+ 65us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 00:58:53.786630 (+ 3417us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33441/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 00:58:53.789256 (+ 2626us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.201.126:43173: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33441/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":2194,"thread_start_us":392,"threads_started":1}
I20260812 00:58:53.791246 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 31518
I20260812 00:58:53.793002 31621 generic_service.cc:196] Checking for leaks (request via RPC)
E20260812 00:58:53.800223 31684 webserver.cc:533] Webserver: sslize failed:
E20260812 00:58:53.800381 31684 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 00:58:53.894731 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 31518
2026-08-12T00:58:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (867 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-12T00:58:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:58:53Z Disabled control of system clock
I20260812 00:58:54.009117 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:39733
--webserver_interface=127.21.201.126
--webserver_port=0
--builtin_ntp_servers=127.21.201.84:44459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:39733
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786496278741276-22309-0/testchainca.cert with env {}
W20260812 00:58:54.382261 31690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:54.382639 31690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:54.382736 31690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:54.391228 31690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:58:54.391435 31690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:54.391479 31690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:54.391541 31690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:58:54.391608 31690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 00:58:54.391672 31690 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 00:58:54.391717 31690 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260812 00:58:54.403717 31690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:44459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:39733
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:39733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786496278741276-22309-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:54.405442 31690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:54.406922 31690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:54.420843 31697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:54.420925 31699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:54.420873 31704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:54.422603 31690 server_base.cc:1034] running on GCE node
I20260812 00:58:54.423614 31690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:54.425450 31690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:54.426749 31690 hybrid_clock.cc:648] HybridClock initialized: now 1786496334426694 us; error 48 us; skew 500 ppm
I20260812 00:58:54.427203 31690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 00:58:54.430171 31690 webserver.cc:460] Webserver started at http://127.21.201.126:39903/ using document root <none> and password file <none>
I20260812 00:58:54.431133 31690 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:54.431309 31690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:54.431672 31690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:54.434950 31690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "ceb92b3dc9f84735844aa1a4b32d002b"
format_stamp: "Formatted at 2026-08-12 00:58:54 on dist-test-slave-7f01"
server_key: "1548980cfc2c363095cdcb8664dd9cfd"
server_key_iv: "0f655b73d80c1d883b2bd5a0c3ff9514"
server_key_version: "encryptionkey@0"
I20260812 00:58:54.436014 31690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "ceb92b3dc9f84735844aa1a4b32d002b"
format_stamp: "Formatted at 2026-08-12 00:58:54 on dist-test-slave-7f01"
server_key: "1548980cfc2c363095cdcb8664dd9cfd"
server_key_iv: "0f655b73d80c1d883b2bd5a0c3ff9514"
server_key_version: "encryptionkey@0"
I20260812 00:58:54.442444 31690 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 00:58:54.448153 31711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:54.449942 31690 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 00:58:54.450343 31690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "ceb92b3dc9f84735844aa1a4b32d002b"
format_stamp: "Formatted at 2026-08-12 00:58:54 on dist-test-slave-7f01"
server_key: "1548980cfc2c363095cdcb8664dd9cfd"
server_key_iv: "0f655b73d80c1d883b2bd5a0c3ff9514"
server_key_version: "encryptionkey@0"
I20260812 00:58:54.450649 31690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:54.471367 31690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:54.472476 31690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:54.472812 31690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:54.494501 31690 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:39733
I20260812 00:58:54.494519 31803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:39733 every 8 connection(s)
I20260812 00:58:54.496070 31690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 00:58:54.498013 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 31690
I20260812 00:58:54.498240 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 00:58:54.498925 22309 external_mini_cluster.cc:1428] Setting key 3f62b226d6061c1abfe7e1ac4ef7b6d7
I20260812 00:58:54.500830 31805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:54.515267 31805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceb92b3dc9f84735844aa1a4b32d002b: Bootstrap starting.
I20260812 00:58:54.516537 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:54.500909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40582 (local address 127.21.201.126:39733)
0812 00:58:54.502184 (+ 1275us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:54.502201 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:54.502273 (+ 72us) server_negotiation.cc:407] Connection header received
0812 00:58:54.504610 (+ 2337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:54.504636 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:54.505410 (+ 774us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:54.505999 (+ 589us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:54.506647 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:54.508785 (+ 2138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:54.510421 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:54.511319 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:54.511627 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:54.513217 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:54.513228 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:54.513249 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:58:54.514062 (+ 813us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:54.514167 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:58:54.514199 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:54.514363 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:54.514656 (+ 293us) server_negotiation.cc:299] Negotiation successful
0812 00:58:54.515097 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":403,"threads_started":1}
I20260812 00:58:54.519115 31805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ceb92b3dc9f84735844aa1a4b32d002b: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:54.520659 31805 log.cc:824] T 00000000000000000000000000000000 P ceb92b3dc9f84735844aa1a4b32d002b: Log is configured to *not* fsync() on all Append() calls
I20260812 00:58:54.522051 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:58:54.524667 31805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceb92b3dc9f84735844aa1a4b32d002b: No bootstrap required, opened a new log
I20260812 00:58:54.535360 31805 raft_consensus.cc:348] T 00000000000000000000000000000000 P ceb92b3dc9f84735844aa1a4b32d002b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb92b3dc9f84735844aa1a4b32d002b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 39733 } }
I20260812 00:58:54.535933 31805 raft_consensus.cc:374] T 00000000000000000000000000000000 P ceb92b3dc9f84735844aa1a4b32d002b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:54.536069 31805 raft_consensus.cc:729] T 00000000000000000000000000000000 P ceb92b3dc9f84735844aa1a4b32d002b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb92b3dc9f84735844aa1a4b32d002b, State: Initialized, Role: FOLLOWER
I20260812 00:58:54.536695 31805 consensus_queue.cc:260] T 00000000000000000000000000000000 P ceb92b3dc9f84735844aa1a4b32d002b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb92b3dc9f84735844aa1a4b32d002b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 39733 } }
I20260812 00:58:54.536985 31805 raft_consensus.cc:388] T 00000000000000000000000000000000 P ceb92b3dc9f84735844aa1a4b32d002b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:58:54.537150 31805 raft_consensus.cc:482] T 00000000000000000000000000000000 P ceb92b3dc9f84735844aa1a4b32d002b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:58:54.537358 31805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ceb92b3dc9f84735844aa1a4b32d002b [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:54.539997 31805 raft_consensus.cc:504] T 00000000000000000000000000000000 P ceb92b3dc9f84735844aa1a4b32d002b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb92b3dc9f84735844aa1a4b32d002b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 39733 } }
I20260812 00:58:54.540575 31805 leader_election.cc:302] T 00000000000000000000000000000000 P ceb92b3dc9f84735844aa1a4b32d002b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ceb92b3dc9f84735844aa1a4b32d002b; no voters:
I20260812 00:58:54.541252 31805 leader_election.cc:288] T 00000000000000000000000000000000 P ceb92b3dc9f84735844aa1a4b32d002b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:58:54.541534 31813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ceb92b3dc9f84735844aa1a4b32d002b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:58:54.542382 31813 raft_consensus.cc:686] T 00000000000000000000000000000000 P ceb92b3dc9f84735844aa1a4b32d002b [term 1 LEADER]: Becoming Leader. State: Replica: ceb92b3dc9f84735844aa1a4b32d002b, State: Running, Role: LEADER
I20260812 00:58:54.542986 31813 consensus_queue.cc:237] T 00000000000000000000000000000000 P ceb92b3dc9f84735844aa1a4b32d002b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb92b3dc9f84735844aa1a4b32d002b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 39733 } }
I20260812 00:58:54.543701 31805 sys_catalog.cc:564] T 00000000000000000000000000000000 P ceb92b3dc9f84735844aa1a4b32d002b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:58:54.546103 31814 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb92b3dc9f84735844aa1a4b32d002b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ceb92b3dc9f84735844aa1a4b32d002b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb92b3dc9f84735844aa1a4b32d002b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 39733 } } }
I20260812 00:58:54.546207 31818 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb92b3dc9f84735844aa1a4b32d002b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ceb92b3dc9f84735844aa1a4b32d002b. Latest consensus state: current_term: 1 leader_uuid: "ceb92b3dc9f84735844aa1a4b32d002b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb92b3dc9f84735844aa1a4b32d002b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 39733 } } }
I20260812 00:58:54.546676 31818 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb92b3dc9f84735844aa1a4b32d002b [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:54.546550 31814 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb92b3dc9f84735844aa1a4b32d002b [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:54.548825 31827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:58:54.554773 31827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:58:54.565883 31827 catalog_manager.cc:1347] Generated new cluster ID: 10f4e284d4ff4c1496619edf10dba88d
I20260812 00:58:54.566058 31827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:58:54.582612 31827 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:58:54.584589 31827 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:58:54.600369 31827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ceb92b3dc9f84735844aa1a4b32d002b: Generated new TSK 0
I20260812 00:58:54.601368 31827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:58:54.640928 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:54.617234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40592 (local address 127.21.201.126:39733)
0812 00:58:54.617491 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:54.617498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:54.617899 (+ 401us) server_negotiation.cc:407] Connection header received
0812 00:58:54.618270 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:54.618275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:54.618353 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:54.618603 (+ 250us) server_negotiation.cc:226] Negotiated authn=JWT
0812 00:58:54.619221 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:54.620095 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:54.622906 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:54.623607 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:54.623828 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:54.624793 (+ 965us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 00:58:54.640215 (+ 15422us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 00:58:54.640643 (+ 428us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40592: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":59}
W20260812 00:58:54.641136 31810 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40592: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260812 00:58:54.642107 31860 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:58:54.617092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:39733 (local address 127.0.0.1:40592)
0812 00:58:54.617778 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:54.617800 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:54.617980 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:54.618647 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:54.618653 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:54.618667 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0812 00:58:54.619003 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:54.619017 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:54.620412 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:54.620420 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:54.622595 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:54.622636 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:54.623918 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:54.623927 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:54.624216 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:54.624291 (+ 75us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 00:58:54.640656 (+ 16365us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 00:58:54.641142 (+ 486us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.201.126:39733: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":363,"threads_started":1}
I20260812 00:58:54.643821 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 31690
I20260812 00:58:54.645195 31797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:58:54.748720 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 31690
2026-08-12T00:58:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (827 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (475 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (462 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:58:55 dist-test-slave-7f01 krb5kdc[31886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:58:55 dist-test-slave-7f01 krb5kdc[31886](info): set up 2 sockets
Aug 12 00:58:55 dist-test-slave-7f01 krb5kdc[31886](info): commencing operation
krb5kdc: starting...
W20260812 00:58:57.128156 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.391s user 0.003s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 00:58:57 dist-test-slave-7f01 krb5kdc[31886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:58:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:58:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 00:58:57.412591 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:33015
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:45707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:33015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:57.781900 31912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:57.782366 31912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:57.782557 31912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:57.792541 31912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:58:57.792770 31912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:57.792907 31912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:57.793025 31912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:58:57.793116 31912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:58:57.804118 31912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:45707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:33015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:33015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:57.805680 31912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:57.807166 31912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:57.820731 31919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:57.820851 31918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:57.820891 31922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:57.821595 31912 server_base.cc:1034] running on GCE node
I20260812 00:58:57.822620 31912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:57.824441 31912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:57.825743 31912 hybrid_clock.cc:648] HybridClock initialized: now 1786496337825676 us; error 53 us; skew 500 ppm
I20260812 00:58:57.826187 31912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:57 dist-test-slave-7f01 krb5kdc[31886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496337, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:57.833709 31912 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 00:58:57.836345 31912 webserver.cc:460] Webserver started at http://127.21.201.126:33651/ using document root <none> and password file <none>
I20260812 00:58:57.837260 31912 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:57.837440 31912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:57.837826 31912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:57.841343 31912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "1526de9ba5d84d6aa34cc1a313e3a47d"
format_stamp: "Formatted at 2026-08-12 00:58:57 on dist-test-slave-7f01"
server_key: "c03eb3d08f08a3806e649f4ef74da80c"
server_key_iv: "bfcbda30956acbf8ca4d0c69196cfbbf"
server_key_version: "encryptionkey@0"
I20260812 00:58:57.842242 31912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "1526de9ba5d84d6aa34cc1a313e3a47d"
format_stamp: "Formatted at 2026-08-12 00:58:57 on dist-test-slave-7f01"
server_key: "c03eb3d08f08a3806e649f4ef74da80c"
server_key_iv: "bfcbda30956acbf8ca4d0c69196cfbbf"
server_key_version: "encryptionkey@0"
I20260812 00:58:57.849118 31912 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 00:58:57.853124 31931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:57.854615 31912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260812 00:58:57.854825 31912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "1526de9ba5d84d6aa34cc1a313e3a47d"
format_stamp: "Formatted at 2026-08-12 00:58:57 on dist-test-slave-7f01"
server_key: "c03eb3d08f08a3806e649f4ef74da80c"
server_key_iv: "bfcbda30956acbf8ca4d0c69196cfbbf"
server_key_version: "encryptionkey@0"
I20260812 00:58:57.855062 31912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:57.867015 31912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:57.871985 31912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:57.872347 31912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:57.895565 31912 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:33015
I20260812 00:58:57.895594 32023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:33015 every 8 connection(s)
I20260812 00:58:57.897588 31912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 00:58:57.901728 32024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:58:57.903947 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 31912
I20260812 00:58:57.904217 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 00:58:57.904942 22309 external_mini_cluster.cc:1428] Setting key ea1499faa52289aa444eb564dd678226
I20260812 00:58:57.916062 32024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1526de9ba5d84d6aa34cc1a313e3a47d: Bootstrap starting.
I20260812 00:58:57.919946 32024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1526de9ba5d84d6aa34cc1a313e3a47d: Neither blocks nor log segments found. Creating new log.
I20260812 00:58:57.921041 32024 log.cc:824] T 00000000000000000000000000000000 P 1526de9ba5d84d6aa34cc1a313e3a47d: Log is configured to *not* fsync() on all Append() calls
Aug 12 00:58:57 dist-test-slave-7f01 krb5kdc[31886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:57.924598 32024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1526de9ba5d84d6aa34cc1a313e3a47d: No bootstrap required, opened a new log
I20260812 00:58:57.932317 32027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:57.907943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58234 (local address 127.21.201.126:33015)
0812 00:58:57.908817 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:57.908834 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:57.908890 (+ 56us) server_negotiation.cc:407] Connection header received
0812 00:58:57.911487 (+ 2597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:57.911522 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:57.912146 (+ 624us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:57.912658 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:57.914244 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:57.916292 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:57.917663 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:57.918749 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:57.918941 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:57.923794 (+ 4853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:57.923812 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:57.923829 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:57.923883 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:57.927495 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:57.928557 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:57.928563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:57.928570 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:57.928749 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:57.929227 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:57.929235 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:57.929237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:57.929859 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:57.930220 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:57.930614 (+ 394us) server_negotiation.cc:299] Negotiation successful
0812 00:58:57.931072 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":377,"threads_started":1}
I20260812 00:58:57.934211 32024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1526de9ba5d84d6aa34cc1a313e3a47d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1526de9ba5d84d6aa34cc1a313e3a47d" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 33015 } }
I20260812 00:58:57.934723 32024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1526de9ba5d84d6aa34cc1a313e3a47d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:58:57.934866 32024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1526de9ba5d84d6aa34cc1a313e3a47d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1526de9ba5d84d6aa34cc1a313e3a47d, State: Initialized, Role: FOLLOWER
I20260812 00:58:57.935519 32024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1526de9ba5d84d6aa34cc1a313e3a47d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1526de9ba5d84d6aa34cc1a313e3a47d" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 33015 } }
I20260812 00:58:57.935811 32024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1526de9ba5d84d6aa34cc1a313e3a47d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:58:57.935987 32024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1526de9ba5d84d6aa34cc1a313e3a47d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:58:57.936170 32024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1526de9ba5d84d6aa34cc1a313e3a47d [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:58:57.938506 32024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1526de9ba5d84d6aa34cc1a313e3a47d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1526de9ba5d84d6aa34cc1a313e3a47d" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 33015 } }
I20260812 00:58:57.939044 32024 leader_election.cc:302] T 00000000000000000000000000000000 P 1526de9ba5d84d6aa34cc1a313e3a47d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1526de9ba5d84d6aa34cc1a313e3a47d; no voters:
I20260812 00:58:57.939728 32024 leader_election.cc:288] T 00000000000000000000000000000000 P 1526de9ba5d84d6aa34cc1a313e3a47d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:58:57.939877 32029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1526de9ba5d84d6aa34cc1a313e3a47d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:58:57.940681 32029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1526de9ba5d84d6aa34cc1a313e3a47d [term 1 LEADER]: Becoming Leader. State: Replica: 1526de9ba5d84d6aa34cc1a313e3a47d, State: Running, Role: LEADER
I20260812 00:58:57.941332 32029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1526de9ba5d84d6aa34cc1a313e3a47d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1526de9ba5d84d6aa34cc1a313e3a47d" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 33015 } }
I20260812 00:58:57.942225 32024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1526de9ba5d84d6aa34cc1a313e3a47d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:58:57.945354 32032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1526de9ba5d84d6aa34cc1a313e3a47d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1526de9ba5d84d6aa34cc1a313e3a47d. Latest consensus state: current_term: 1 leader_uuid: "1526de9ba5d84d6aa34cc1a313e3a47d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1526de9ba5d84d6aa34cc1a313e3a47d" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 33015 } } }
I20260812 00:58:57.945760 32032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1526de9ba5d84d6aa34cc1a313e3a47d [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:57.946461 32030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1526de9ba5d84d6aa34cc1a313e3a47d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1526de9ba5d84d6aa34cc1a313e3a47d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1526de9ba5d84d6aa34cc1a313e3a47d" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 33015 } } }
I20260812 00:58:57.946789 32030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1526de9ba5d84d6aa34cc1a313e3a47d [sys.catalog]: This master's current role is: LEADER
I20260812 00:58:57.946798 32043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:58:57.955363 32043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:58:57.970433 32043 catalog_manager.cc:1347] Generated new cluster ID: 11fd05511d07405a8190b48caa11fc0d
I20260812 00:58:57.970593 32043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:58:58.010360 32043 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:58:58.011881 32043 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:58:58.019527 32043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1526de9ba5d84d6aa34cc1a313e3a47d: Generated new TSK 0
I20260812 00:58:58.020458 32043 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.201.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
I20260812 00:58:58.088898 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:0
--local_ip_for_outbound_sockets=127.21.201.65
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:33015
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:45707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:58.472620 32071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:58.472988 32071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:58.473171 32071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:58.481048 32071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:58.481215 32071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:58.481424 32071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 00:58:58.492336 32071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:45707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:33015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:58.493834 32071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:58.495258 32071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:58.514261 32085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:58.514429 32084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:58.519858 32087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:58.520956 32071 server_base.cc:1034] running on GCE node
I20260812 00:58:58.521842 32071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:58.523099 32071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:58.524348 32071 hybrid_clock.cc:648] HybridClock initialized: now 1786496338524288 us; error 83 us; skew 500 ppm
I20260812 00:58:58.524740 32071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:58 dist-test-slave-7f01 krb5kdc[31886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496338, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:58.529112 32071 init.cc:377] Logged in from keytab as kudu/127.21.201.65@KRBTEST.COM (short username kudu)
I20260812 00:58:58.531666 32071 webserver.cc:460] Webserver started at http://127.21.201.65:44799/ using document root <none> and password file <none>
I20260812 00:58:58.532611 32071 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:58.532788 32071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:58.533154 32071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:58.536827 32071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-0/data/instance:
uuid: "a31cc44171fc4ee1aa4a00e1fde65e75"
format_stamp: "Formatted at 2026-08-12 00:58:58 on dist-test-slave-7f01"
server_key: "2d8fa73d61d636801f6734fba962171c"
server_key_iv: "9a932b6fda64364d4505bb0d2d9008b9"
server_key_version: "encryptionkey@0"
I20260812 00:58:58.537773 32071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance:
uuid: "a31cc44171fc4ee1aa4a00e1fde65e75"
format_stamp: "Formatted at 2026-08-12 00:58:58 on dist-test-slave-7f01"
server_key: "2d8fa73d61d636801f6734fba962171c"
server_key_iv: "9a932b6fda64364d4505bb0d2d9008b9"
server_key_version: "encryptionkey@0"
I20260812 00:58:58.543992 32071 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260812 00:58:58.547904 32102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:58.549510 32071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 00:58:58.549670 32071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "a31cc44171fc4ee1aa4a00e1fde65e75"
format_stamp: "Formatted at 2026-08-12 00:58:58 on dist-test-slave-7f01"
server_key: "2d8fa73d61d636801f6734fba962171c"
server_key_iv: "9a932b6fda64364d4505bb0d2d9008b9"
server_key_version: "encryptionkey@0"
I20260812 00:58:58.549928 32071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:58.582333 32071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:58.588661 32071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:58.589043 32071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:58.590281 32071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:58:58.591854 32071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:58.591936 32071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:58.592089 32071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:58.592162 32071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:58.632511 32071 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:37697
I20260812 00:58:58.632531 32272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:37697 every 8 connection(s)
I20260812 00:58:58.634200 32071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 00:58:58.639099 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 32071
I20260812 00:58:58.639423 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance
I20260812 00:58:58.640547 22309 external_mini_cluster.cc:1428] Setting key 07a58d174bfc1caa354d1ed183483d36
Aug 12 00:58:58 dist-test-slave-7f01 krb5kdc[31886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496338, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:58.665813 32276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:58.637665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:56185 (local address 127.21.201.126:33015)
0812 00:58:58.638515 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:58.638524 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:58.641056 (+ 2532us) server_negotiation.cc:407] Connection header received
0812 00:58:58.643868 (+ 2812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:58.643874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:58.643980 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:58.644106 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:58.646593 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:58.647926 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:58.650955 (+ 3029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:58.651694 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:58.651892 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:58.657532 (+ 5640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:58.657540 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:58.657544 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:58.657588 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:58.661601 (+ 4013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:58.662937 (+ 1336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:58.662943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:58.662947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:58.663127 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:58.663758 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:58.663763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:58.663767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:58.664126 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:58.664290 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:58.665308 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0812 00:58:58.665523 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":440,"threads_started":1}
I20260812 00:58:58.667596 32277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:58.637945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:33015 (local address 127.21.201.65:56185)
0812 00:58:58.640687 (+ 2742us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:58.640870 (+ 183us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:58.643505 (+ 2635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:58.644438 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:58.644450 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:58.645352 (+ 902us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:58.646331 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:58.646363 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:58.648142 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:58.648151 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:58.650535 (+ 2384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:58.650557 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:58.652111 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:58.652123 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:58.652749 (+ 626us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:58.653731 (+ 982us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:58.653769 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:58.657161 (+ 3392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:58.661797 (+ 4636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:58.661812 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:58.661834 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:58.662792 (+ 958us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:58.663268 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:58.663275 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:58.663282 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:58.663606 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:58.664303 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:58.664315 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:58.664846 (+ 531us) client_negotiation.cc:769] Sending connection context
0812 00:58:58.665213 (+ 367us) client_negotiation.cc:240] Negotiation successful
0812 00:58:58.665916 (+ 703us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2268,"thread_start_us":1948,"threads_started":1}
I20260812 00:58:58.669584 32274 heartbeater.cc:347] Connected to a master server at 127.21.201.126:33015
I20260812 00:58:58.669950 32274 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:58.671156 32274 heartbeater.cc:511] Master 127.21.201.126:33015 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.201.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.66@KRBTEST.COM" created.
I20260812 00:58:58.674273 31963 ts_manager.cc:194] Registered new tserver with Master: a31cc44171fc4ee1aa4a00e1fde65e75 (127.21.201.65:37697)
I20260812 00:58:58.676729 31963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.65@KRBTEST.COM'} at 127.21.201.65:56185
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
I20260812 00:58:58.708294 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:0
--local_ip_for_outbound_sockets=127.21.201.66
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:33015
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:45707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:59.093410 32282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:59.093832 32282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:59.094029 32282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:59.102473 32282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:59.102660 32282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:59.102850 32282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 00:58:59.114935 32282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:45707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:33015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:59.116603 32282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:59.118170 32282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:59.134424 32291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:59.134354 32296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:59.134444 32293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:59.135715 32282 server_base.cc:1034] running on GCE node
I20260812 00:58:59.136824 32282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:59.138242 32282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:59.139571 32282 hybrid_clock.cc:648] HybridClock initialized: now 1786496339139488 us; error 80 us; skew 500 ppm
I20260812 00:58:59.140100 32282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:59 dist-test-slave-7f01 krb5kdc[31886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496339, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:59.144614 32282 init.cc:377] Logged in from keytab as kudu/127.21.201.66@KRBTEST.COM (short username kudu)
I20260812 00:58:59.146539 32282 webserver.cc:460] Webserver started at http://127.21.201.66:42703/ using document root <none> and password file <none>
I20260812 00:58:59.147321 32282 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:59.147495 32282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:59.147922 32282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:59.151177 32282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-1/data/instance:
uuid: "4ffc65aecce64689b3c04ed2c59b004f"
format_stamp: "Formatted at 2026-08-12 00:58:59 on dist-test-slave-7f01"
server_key: "2547fbdb74c3d7cd9267eee5810ddbd4"
server_key_iv: "72ca95dec78e697648cf1d3df8097a8f"
server_key_version: "encryptionkey@0"
I20260812 00:58:59.152141 32282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance:
uuid: "4ffc65aecce64689b3c04ed2c59b004f"
format_stamp: "Formatted at 2026-08-12 00:58:59 on dist-test-slave-7f01"
server_key: "2547fbdb74c3d7cd9267eee5810ddbd4"
server_key_iv: "72ca95dec78e697648cf1d3df8097a8f"
server_key_version: "encryptionkey@0"
I20260812 00:58:59.158541 32282 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 00:58:59.163439 32308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:59.165071 32282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 00:58:59.165234 32282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "4ffc65aecce64689b3c04ed2c59b004f"
format_stamp: "Formatted at 2026-08-12 00:58:59 on dist-test-slave-7f01"
server_key: "2547fbdb74c3d7cd9267eee5810ddbd4"
server_key_iv: "72ca95dec78e697648cf1d3df8097a8f"
server_key_version: "encryptionkey@0"
I20260812 00:58:59.165452 32282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:59.178254 32282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:59.182989 32282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:59.183266 32282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:59.184358 32282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:58:59.185806 32282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:59.185890 32282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:59.185981 32282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:59.186025 32282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:59.222961 32282 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:45243
I20260812 00:58:59.223011 32485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:45243 every 8 connection(s)
I20260812 00:58:59.224457 32282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 00:58:59.230726 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 32282
I20260812 00:58:59.230952 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance
I20260812 00:58:59.231684 22309 external_mini_cluster.cc:1428] Setting key 0f6dd1f15ee9fde7b84dc4cfab27f1fe
Aug 12 00:58:59 dist-test-slave-7f01 krb5kdc[31886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496339, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:59.250074 32488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:59.227521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:46881 (local address 127.21.201.126:33015)
0812 00:58:59.228092 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:59.228099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:59.229259 (+ 1160us) server_negotiation.cc:407] Connection header received
0812 00:58:59.231426 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:59.231433 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:59.231538 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:59.231677 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:59.234486 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:59.235670 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:59.237558 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:59.238296 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:59.238447 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:59.242668 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:59.242678 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:59.242686 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:59.242736 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:59.246207 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:59.247340 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:59.247344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:59.247347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:59.247478 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:59.248061 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:59.248065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:59.248067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:59.248372 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:59.248499 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:59.249577 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0812 00:58:59.249774 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":286,"threads_started":1}
I20260812 00:58:59.251711 32489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:59.228099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:33015 (local address 127.21.201.66:46881)
0812 00:58:59.229077 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:59.229121 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:59.230834 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:59.232085 (+ 1251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:59.232103 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:59.232869 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:59.234194 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:59.234232 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:59.235948 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:59.235953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:59.237370 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:59.237397 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:59.238543 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:59.238548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:59.238764 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:59.239677 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:59.239703 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:59.242376 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:59.246407 (+ 4031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:59.246421 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:59.246443 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:59.247175 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:59.247606 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:59.247614 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:59.247621 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:59.247921 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:59.248515 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:59.248527 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:59.249073 (+ 546us) client_negotiation.cc:769] Sending connection context
0812 00:58:59.249439 (+ 366us) client_negotiation.cc:240] Negotiation successful
0812 00:58:59.250068 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":451,"threads_started":1}
I20260812 00:58:59.253540 32486 heartbeater.cc:347] Connected to a master server at 127.21.201.126:33015
I20260812 00:58:59.253969 32486 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:59.254739 32486 heartbeater.cc:511] Master 127.21.201.126:33015 requested a full tablet report, sending...
I20260812 00:58:59.257232 31963 ts_manager.cc:194] Registered new tserver with Master: 4ffc65aecce64689b3c04ed2c59b004f (127.21.201.66:45243)
I20260812 00:58:59.258096 31963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.66@KRBTEST.COM'} at 127.21.201.66:46881
WARNING: no policy specified for kudu/127.21.201.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
I20260812 00:58:59.295512 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:0
--local_ip_for_outbound_sockets=127.21.201.67
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:33015
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:45707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:58:59.671569 32493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:58:59.672013 32493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:58:59.672199 32493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:58:59.680048 32493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:58:59.680244 32493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:58:59.680436 32493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 00:58:59.681316 32274 heartbeater.cc:503] Master 127.21.201.126:33015 was elected leader, sending a full tablet report...
I20260812 00:58:59.691180 32493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:45707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:33015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:58:59.692764 32493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:58:59.694319 32493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:58:59.708869 32500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:59.708963 32501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:58:59.708973 32504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:58:59.709766 32493 server_base.cc:1034] running on GCE node
I20260812 00:58:59.710536 32493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:58:59.712159 32493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:58:59.713452 32493 hybrid_clock.cc:648] HybridClock initialized: now 1786496339713407 us; error 139 us; skew 500 ppm
I20260812 00:58:59.713897 32493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:58:59 dist-test-slave-7f01 krb5kdc[31886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496339, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:59.717537 32493 init.cc:377] Logged in from keytab as kudu/127.21.201.67@KRBTEST.COM (short username kudu)
I20260812 00:58:59.719280 32493 webserver.cc:460] Webserver started at http://127.21.201.67:33409/ using document root <none> and password file <none>
I20260812 00:58:59.720136 32493 fs_manager.cc:359] Metadata directory not provided
I20260812 00:58:59.720304 32493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:58:59.720643 32493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:58:59.724156 32493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-2/data/instance:
uuid: "3e17790296494f289121c3d359758357"
format_stamp: "Formatted at 2026-08-12 00:58:59 on dist-test-slave-7f01"
server_key: "667d797068871c866b0a129db72bc30f"
server_key_iv: "c51e64bfaefc3bb1456f6ac076896c95"
server_key_version: "encryptionkey@0"
I20260812 00:58:59.725093 32493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance:
uuid: "3e17790296494f289121c3d359758357"
format_stamp: "Formatted at 2026-08-12 00:58:59 on dist-test-slave-7f01"
server_key: "667d797068871c866b0a129db72bc30f"
server_key_iv: "c51e64bfaefc3bb1456f6ac076896c95"
server_key_version: "encryptionkey@0"
I20260812 00:58:59.731056 32493 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 00:58:59.736107 32519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:59.737766 32493 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 00:58:59.737999 32493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "3e17790296494f289121c3d359758357"
format_stamp: "Formatted at 2026-08-12 00:58:59 on dist-test-slave-7f01"
server_key: "667d797068871c866b0a129db72bc30f"
server_key_iv: "c51e64bfaefc3bb1456f6ac076896c95"
server_key_version: "encryptionkey@0"
I20260812 00:58:59.738257 32493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:58:59.755036 32493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:58:59.760262 32493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:58:59.760617 32493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:58:59.761768 32493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:58:59.763288 32493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:58:59.763396 32493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:59.763554 32493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:58:59.763638 32493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:58:59.802804 32493 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:44517
I20260812 00:58:59.802850 32703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:44517 every 8 connection(s)
I20260812 00:58:59.804317 32493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 00:58:59.806092 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 32493
I20260812 00:58:59.806291 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance
I20260812 00:58:59.806881 22309 external_mini_cluster.cc:1428] Setting key 4c57535a42ad36ac412038b79d01e925
Aug 12 00:58:59 dist-test-slave-7f01 krb5kdc[31886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496339, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:58:59.827229 32707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:59.807458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:52149 (local address 127.21.201.126:33015)
0812 00:58:59.808018 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:59.808023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:59.811196 (+ 3173us) server_negotiation.cc:407] Connection header received
0812 00:58:59.811252 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:59.811255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:59.811325 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:59.811408 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:59.813161 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:59.813963 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:59.815682 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:59.816226 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:59.816345 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:59.820475 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:59.820481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:59.820484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:59.820528 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:59.823829 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:59.824648 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:59.824655 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:59.824664 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:59.824794 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:59.825284 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:59.825288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:59.825290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:59.825606 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:59.825772 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:59.826549 (+ 777us) server_negotiation.cc:299] Negotiation successful
0812 00:58:59.826823 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":286,"threads_started":1}
I20260812 00:58:59.828344 32708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:59.808111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:33015 (local address 127.21.201.67:52149)
0812 00:58:59.809074 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0812 00:58:59.809125 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 00:58:59.810905 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:58:59.811669 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:58:59.811678 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:58:59.812357 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:58:59.812982 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:59.813002 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:59.814154 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:59.814159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:59.815500 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:58:59.815514 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:58:59.816491 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:58:59.816496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:58:59.816724 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:59.817494 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:58:59.817519 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:58:59.820187 (+ 2668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:58:59.824004 (+ 3817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:59.824014 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:59.824039 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:59.824508 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:59.824915 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:58:59.824920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:58:59.824924 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:58:59.825145 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:58:59.825745 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:58:59.825754 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:58:59.826143 (+ 389us) client_negotiation.cc:769] Sending connection context
0812 00:58:59.826420 (+ 277us) client_negotiation.cc:240] Negotiation successful
0812 00:58:59.827064 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":439,"threads_started":1}
I20260812 00:58:59.829831 32704 heartbeater.cc:347] Connected to a master server at 127.21.201.126:33015
I20260812 00:58:59.830152 32704 heartbeater.cc:464] Registering TS with master...
I20260812 00:58:59.830880 32704 heartbeater.cc:511] Master 127.21.201.126:33015 requested a full tablet report, sending...
I20260812 00:58:59.832500 31963 ts_manager.cc:194] Registered new tserver with Master: 3e17790296494f289121c3d359758357 (127.21.201.67:44517)
I20260812 00:58:59.833559 31963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.67@KRBTEST.COM'} at 127.21.201.67:52149
I20260812 00:58:59.835678 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/test-admin.keytab.
Aug 12 00:58:59 dist-test-slave-7f01 krb5kdc[31886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:59.870522 22309 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260812 00:58:59.877571 32718 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:58:59.873978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58600 (local address 127.21.201.65:37697)
0812 00:58:59.875136 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:59.875148 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:59.875186 (+ 38us) server_negotiation.cc:407] Connection header received
0812 00:58:59.875265 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:59.875272 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:59.875514 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:59.875696 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:59.877065 (+ 1369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58600: BlockingRecv error: recv got EOF from 127.0.0.1:58600 (error 108)
Metrics: {"server-negotiator.queue_time_us":935,"thread_start_us":547,"threads_started":1}
W20260812 00:58:59.877846 22309 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/krb5cc' of type FILE
Aug 12 00:58:59 dist-test-slave-7f01 krb5kdc[31886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:59.880709 22309 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 00:58:59 dist-test-slave-7f01 krb5kdc[31886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:59.882858 22309 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 00:58:59 dist-test-slave-7f01 krb5kdc[31886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.65@KRBTEST.COM
I20260812 00:58:59.903944 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:59.885363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58604 (local address 127.21.201.65:37697)
0812 00:58:59.885714 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:59.885722 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:59.886193 (+ 471us) server_negotiation.cc:407] Connection header received
0812 00:58:59.886296 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:59.886303 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:59.886386 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:59.886523 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:59.887868 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:59.888889 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:59.890194 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:59.890812 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:59.890956 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:59.895841 (+ 4885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:59.895868 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:59.895890 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:59.895960 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:59.899688 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:59.900975 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:59.900987 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:59.900990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:59.901198 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:59.901677 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:59.901689 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:59.901691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:59.902204 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:59.902609 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:59.903202 (+ 593us) server_negotiation.cc:299] Negotiation successful
0812 00:58:59.903544 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
W20260812 00:58:59.912580 32718 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:58:59.909789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58618 (local address 127.21.201.65:37697)
0812 00:58:59.910208 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:59.910216 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:59.910537 (+ 321us) server_negotiation.cc:407] Connection header received
0812 00:58:59.910782 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:59.910792 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:59.910872 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:59.910993 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:59.912226 (+ 1233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58618: BlockingRecv error: recv got EOF from 127.0.0.1:58618 (error 108)
Metrics: {"server-negotiator.queue_time_us":195}
W20260812 00:58:59.912815 22309 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/krb5cc' of type FILE
Aug 12 00:58:59 dist-test-slave-7f01 krb5kdc[31886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:59.914989 22309 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 00:58:59 dist-test-slave-7f01 krb5kdc[31886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:59.916919 22309 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 00:58:59 dist-test-slave-7f01 krb5kdc[31886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.65@KRBTEST.COM
I20260812 00:58:59.934625 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:59.918889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58622 (local address 127.21.201.65:37697)
0812 00:58:59.919341 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:59.919350 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:59.919777 (+ 427us) server_negotiation.cc:407] Connection header received
0812 00:58:59.920007 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:59.920012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:59.920099 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:59.920225 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:59.921440 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:59.922398 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:59.924011 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:59.924548 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:59.924737 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:59.929133 (+ 4396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:59.929138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:59.929142 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:59.929185 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:59.931838 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:59.932601 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:59.932609 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:59.932617 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:59.932755 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:59.933152 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:59.933157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:59.933159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:59.933441 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:59.933738 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:59.934091 (+ 353us) server_negotiation.cc:299] Negotiation successful
0812 00:58:59.934311 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
W20260812 00:58:59.939647 32718 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:58:59.937835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58636 (local address 127.21.201.65:37697)
0812 00:58:59.938097 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:59.938103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:59.938261 (+ 158us) server_negotiation.cc:407] Connection header received
0812 00:58:59.938407 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:59.938411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:59.938488 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:59.938570 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:59.939398 (+ 828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58636: BlockingRecv error: recv got EOF from 127.0.0.1:58636 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
W20260812 00:58:59.940138 22309 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786496278741276-22309-0/krb5kdc/krb5cc)
Aug 12 00:58:59 dist-test-slave-7f01 krb5kdc[31886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:59.942022 22309 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 00:58:59 dist-test-slave-7f01 krb5kdc[31886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:58:59.943518 22309 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 00:58:59 dist-test-slave-7f01 krb5kdc[31886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.65@KRBTEST.COM
I20260812 00:58:59.961686 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:59.945240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58652 (local address 127.21.201.65:37697)
0812 00:58:59.945553 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:59.945560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:59.945824 (+ 264us) server_negotiation.cc:407] Connection header received
0812 00:58:59.946086 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:59.946092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:59.946170 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:59.946292 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:59.947569 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:59.948476 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:59.949909 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:59.950379 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:59.950535 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:59.955159 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:59.955166 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:59.955169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:59.955212 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:59.958482 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:59.959415 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:59.959422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:59.959425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:59.959820 (+ 395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:59.960271 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:59.960276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:59.960278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:59.960588 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:59.960885 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:59.961079 (+ 194us) server_negotiation.cc:299] Negotiation successful
0812 00:58:59.961280 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260812 00:58:59.964426 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 32071
I20260812 00:58:59.979272 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:58:59.965525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58662 (local address 127.21.201.65:37697)
0812 00:58:59.965759 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0812 00:58:59.965764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:58:59.965822 (+ 58us) server_negotiation.cc:407] Connection header received
0812 00:58:59.966064 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:58:59.966070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:58:59.966147 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:58:59.966237 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:58:59.967313 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:59.968109 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:59.969438 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:58:59.969973 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:58:59.970078 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:58:59.972489 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:58:59.972495 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:58:59.972498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:58:59.972536 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:58:59.975853 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:59.976638 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:59.976643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:59.976645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:59.976793 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:58:59.977479 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:58:59.977483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:58:59.977485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:58:59.977796 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:58:59.978058 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0812 00:58:59.978550 (+ 492us) server_negotiation.cc:299] Negotiation successful
0812 00:58:59.978758 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260812 00:58:59.980115 32267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:00.069626 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 32071
I20260812 00:59:00.100557 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 32282
Aug 12 00:59:00 dist-test-slave-7f01 krb5kdc[31886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.66@KRBTEST.COM
I20260812 00:59:00.121673 32739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:00.101933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53924 (local address 127.21.201.66:45243)
0812 00:59:00.102675 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:00.102684 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:00.102714 (+ 30us) server_negotiation.cc:407] Connection header received
0812 00:59:00.102785 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:00.102793 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:00.103023 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:00.103244 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:00.104896 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:00.106215 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:00.107875 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:00.108515 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:00.108691 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:00.113107 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:00.113123 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:00.113141 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:00.113184 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:00.117603 (+ 4419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:00.118964 (+ 1361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:00.118975 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:00.118979 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:00.119140 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:00.119603 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:00.119608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:00.119612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:00.120123 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:00.120626 (+ 503us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:00.121000 (+ 374us) server_negotiation.cc:299] Negotiation successful
0812 00:59:00.121314 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":427,"threads_started":1}
I20260812 00:59:00.123334 32476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:00.226341 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 32282
I20260812 00:59:00.255970 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 32493
Aug 12 00:59:00 dist-test-slave-7f01 krb5kdc[31886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.67@KRBTEST.COM
I20260812 00:59:00.274199 32743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:00.257217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42906 (local address 127.21.201.67:44517)
0812 00:59:00.257797 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:00.257804 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:00.257837 (+ 33us) server_negotiation.cc:407] Connection header received
0812 00:59:00.257908 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:00.257915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:00.258144 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:00.258349 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:00.259535 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:00.260761 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:00.262233 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:00.262798 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:00.262965 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:00.267855 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:00.267869 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:00.267896 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:00.267942 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:00.271077 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:00.271965 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:00.271976 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:00.271984 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:00.272114 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:00.272517 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:00.272523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:00.272526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:00.272915 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:00.273254 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:00.273548 (+ 294us) server_negotiation.cc:299] Negotiation successful
0812 00:59:00.273829 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":307,"threads_started":1}
I20260812 00:59:00.276096 32698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:00.367914 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 32493
I20260812 00:59:00.397600 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 31912
I20260812 00:59:00.398975 32016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:00.510939 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 31912
2026-08-12T00:59:00Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4829 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:59:00 dist-test-slave-7f01 krb5kdc[32752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:59:00 dist-test-slave-7f01 krb5kdc[32752](info): set up 2 sockets
Aug 12 00:59:00 dist-test-slave-7f01 krb5kdc[32752](info): commencing operation
krb5kdc: starting...
W20260812 00:59:01.930395 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.363s user 0.002s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 00:59:02 dist-test-slave-7f01 krb5kdc[32752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:59:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:59:02Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.126.keytab.
Entry for principal oryx/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.126.keytab.
I20260812 00:59:02.224534 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:44537
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.126.keytab
--principal=oryx/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:41999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:44537
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:59:02.601766 303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:02.602128 303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:02.602300 303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:02.609932 303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:59:02.610069 303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:02.610119 303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:02.610220 303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:59:02.610296 303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:59:02.620826 303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:41999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:44537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.126.keytab
--principal=oryx/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:44537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:02.622416 303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:02.623937 303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:02.638134 310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:02.638171 315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:02.638074 313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:02.638916 303 server_base.cc:1034] running on GCE node
I20260812 00:59:02.639971 303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:02.641742 303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:02.642993 303 hybrid_clock.cc:648] HybridClock initialized: now 1786496342642911 us; error 58 us; skew 500 ppm
I20260812 00:59:02.643414 303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:02 dist-test-slave-7f01 krb5kdc[32752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496342, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:02.647161 303 init.cc:377] Logged in from keytab as oryx/127.21.201.126@KRBTEST.COM (short username oryx)
I20260812 00:59:02.649142 303 webserver.cc:460] Webserver started at http://127.21.201.126:42227/ using document root <none> and password file <none>
I20260812 00:59:02.649885 303 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:02.650003 303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:02.650266 303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:59:02.653656 303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "3ef13f6098fe4535af6f2557d272fd2b"
format_stamp: "Formatted at 2026-08-12 00:59:02 on dist-test-slave-7f01"
server_key: "37d7d6b22575d3af72237610b68daeae"
server_key_iv: "9e3130ff2d6dbd968dd4381d34f7fce1"
server_key_version: "encryptionkey@0"
I20260812 00:59:02.654565 303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "3ef13f6098fe4535af6f2557d272fd2b"
format_stamp: "Formatted at 2026-08-12 00:59:02 on dist-test-slave-7f01"
server_key: "37d7d6b22575d3af72237610b68daeae"
server_key_iv: "9e3130ff2d6dbd968dd4381d34f7fce1"
server_key_version: "encryptionkey@0"
I20260812 00:59:02.660543 303 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 00:59:02.665853 326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:02.668231 303 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.004s
I20260812 00:59:02.668509 303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "3ef13f6098fe4535af6f2557d272fd2b"
format_stamp: "Formatted at 2026-08-12 00:59:02 on dist-test-slave-7f01"
server_key: "37d7d6b22575d3af72237610b68daeae"
server_key_iv: "9e3130ff2d6dbd968dd4381d34f7fce1"
server_key_version: "encryptionkey@0"
I20260812 00:59:02.668812 303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:02.701275 303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:02.706283 303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:02.706651 303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:02.727502 303 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:44537
I20260812 00:59:02.727815 405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:44537 every 8 connection(s)
I20260812 00:59:02.729048 303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 00:59:02.732934 408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:02.736824 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 303
I20260812 00:59:02.737097 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 00:59:02.737805 22309 external_mini_cluster.cc:1428] Setting key 1dfdfc980f5ff98558095c3a9ca78484
I20260812 00:59:02.747402 408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b: Bootstrap starting.
I20260812 00:59:02.751724 408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:02.753249 408 log.cc:824] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b: Log is configured to *not* fsync() on all Append() calls
Aug 12 00:59:02 dist-test-slave-7f01 krb5kdc[32752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM
I20260812 00:59:02.757532 408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b: No bootstrap required, opened a new log
I20260812 00:59:02.765391 412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:02.740118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36668 (local address 127.21.201.126:44537)
0812 00:59:02.741029 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:02.741042 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:02.741087 (+ 45us) server_negotiation.cc:407] Connection header received
0812 00:59:02.742800 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:02.742823 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:02.743289 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:02.743653 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:02.745769 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:02.747144 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:02.749006 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:02.749711 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:02.749855 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:02.755986 (+ 6131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:02.755998 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:02.756028 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:02.756081 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:02.760424 (+ 4343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:02.761401 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:02.761408 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:02.761416 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:02.761638 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:02.762083 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:02.762087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:02.762089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:02.762800 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:02.763220 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:02.763660 (+ 440us) server_negotiation.cc:299] Negotiation successful
0812 00:59:02.764215 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":429,"threads_started":1}
I20260812 00:59:02.767580 408 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } }
I20260812 00:59:02.768123 408 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:02.768234 408 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef13f6098fe4535af6f2557d272fd2b, State: Initialized, Role: FOLLOWER
I20260812 00:59:02.768911 408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } }
I20260812 00:59:02.769212 408 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:59:02.769312 408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:59:02.769449 408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:02.771826 408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } }
I20260812 00:59:02.772356 408 leader_election.cc:302] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ef13f6098fe4535af6f2557d272fd2b; no voters:
I20260812 00:59:02.773173 408 leader_election.cc:288] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:59:02.773329 416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:59:02.774101 416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [term 1 LEADER]: Becoming Leader. State: Replica: 3ef13f6098fe4535af6f2557d272fd2b, State: Running, Role: LEADER
I20260812 00:59:02.774714 416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } }
I20260812 00:59:02.775893 408 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:59:02.777648 420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ef13f6098fe4535af6f2557d272fd2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } }
I20260812 00:59:02.778143 420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [sys.catalog]: This master's current role is: LEADER
I20260812 00:59:02.778143 419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ef13f6098fe4535af6f2557d272fd2b. Latest consensus state: current_term: 1 leader_uuid: "3ef13f6098fe4535af6f2557d272fd2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } }
I20260812 00:59:02.778544 419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [sys.catalog]: This master's current role is: LEADER
I20260812 00:59:02.783188 425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:59:02.791519 425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:59:02.802621 425 catalog_manager.cc:1347] Generated new cluster ID: d4fe47a674c540db8f1ec527af6061d8
I20260812 00:59:02.802769 425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:59:02.827672 425 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:59:02.829151 425 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:59:02.843480 425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b: Generated new TSK 0
I20260812 00:59:02.844508 425 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.201.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.201.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.201.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.65.keytab.
Entry for principal oryx/127.21.201.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.65.keytab.
I20260812 00:59:02.920368 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:0
--local_ip_for_outbound_sockets=127.21.201.65
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:44537
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.65.keytab
--principal=oryx/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:41999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:59:03.296247 447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:03.296574 447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:03.296752 447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:03.304415 447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:03.304559 447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:03.304785 447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 00:59:03.315129 447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:41999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.65.keytab
--principal=oryx/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:44537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:03.316653 447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:03.318055 447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:03.333864 457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:03.333994 454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:03.333981 455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:03.335383 447 server_base.cc:1034] running on GCE node
I20260812 00:59:03.336246 447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:03.337409 447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:03.338694 447 hybrid_clock.cc:648] HybridClock initialized: now 1786496343338631 us; error 67 us; skew 500 ppm
I20260812 00:59:03.339093 447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:03 dist-test-slave-7f01 krb5kdc[32752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496343, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:03.346277 447 init.cc:377] Logged in from keytab as oryx/127.21.201.65@KRBTEST.COM (short username oryx)
I20260812 00:59:03.348042 447 webserver.cc:460] Webserver started at http://127.21.201.65:34085/ using document root <none> and password file <none>
I20260812 00:59:03.348888 447 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:03.349277 447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:03.349684 447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:59:03.352991 447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/data/instance:
uuid: "2ac902d212d8411bb6adfa954008ed8e"
format_stamp: "Formatted at 2026-08-12 00:59:03 on dist-test-slave-7f01"
server_key: "5187f392bdf1b733b59334ebacb840b3"
server_key_iv: "1f9d72d4a279f22041c4922844e8d153"
server_key_version: "encryptionkey@0"
I20260812 00:59:03.353847 447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance:
uuid: "2ac902d212d8411bb6adfa954008ed8e"
format_stamp: "Formatted at 2026-08-12 00:59:03 on dist-test-slave-7f01"
server_key: "5187f392bdf1b733b59334ebacb840b3"
server_key_iv: "1f9d72d4a279f22041c4922844e8d153"
server_key_version: "encryptionkey@0"
I20260812 00:59:03.359722 447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260812 00:59:03.363672 471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:03.365331 447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 00:59:03.365527 447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "2ac902d212d8411bb6adfa954008ed8e"
format_stamp: "Formatted at 2026-08-12 00:59:03 on dist-test-slave-7f01"
server_key: "5187f392bdf1b733b59334ebacb840b3"
server_key_iv: "1f9d72d4a279f22041c4922844e8d153"
server_key_version: "encryptionkey@0"
I20260812 00:59:03.365773 447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:03.383081 447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:03.388186 447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:03.388527 447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:03.389717 447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:59:03.391237 447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:59:03.391346 447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:03.391511 447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:59:03.391590 447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:03.431216 447 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:37425
I20260812 00:59:03.431265 658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:37425 every 8 connection(s)
I20260812 00:59:03.432679 447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 00:59:03.441113 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 447
I20260812 00:59:03.441399 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance
I20260812 00:59:03.442094 22309 external_mini_cluster.cc:1428] Setting key 7badd9b897db9d199fb91ec186926a99
Aug 12 00:59:03 dist-test-slave-7f01 krb5kdc[32752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496343, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM
I20260812 00:59:03.456048 661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:03.435527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:44351 (local address 127.21.201.126:44537)
0812 00:59:03.436324 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:03.436330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:03.437508 (+ 1178us) server_negotiation.cc:407] Connection header received
0812 00:59:03.439687 (+ 2179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:03.439693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:03.439826 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:03.439992 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:03.441715 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:03.442553 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:03.444526 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:03.444954 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:03.445096 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:03.449095 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:03.449101 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:03.449105 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:03.449140 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:03.452211 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:03.453040 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:03.453044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:03.453047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:03.453172 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:03.453687 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:03.453691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:03.453693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:03.453999 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:03.454123 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:03.455231 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0812 00:59:03.455530 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":403,"threads_started":1}
I20260812 00:59:03.457715 662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:03.436198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.21.201.65:44351)
0812 00:59:03.437305 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:03.437348 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:03.439365 (+ 2017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:03.440245 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:03.440254 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:03.440850 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:03.441528 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:03.441558 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:03.442732 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:03.442742 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:03.444367 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:03.444381 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:03.445211 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:03.445219 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:03.445477 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:03.446273 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:03.446298 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:03.448819 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:03.452383 (+ 3564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:03.452392 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:03.452419 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:03.452883 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:03.453292 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:03.453297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:03.453301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:03.453555 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:03.454137 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:03.454147 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:03.454668 (+ 521us) client_negotiation.cc:769] Sending connection context
0812 00:59:03.455048 (+ 380us) client_negotiation.cc:240] Negotiation successful
0812 00:59:03.456101 (+ 1053us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":420,"threads_started":1}
I20260812 00:59:03.459896 659 heartbeater.cc:347] Connected to a master server at 127.21.201.126:44537
I20260812 00:59:03.460278 659 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:03.461135 659 heartbeater.cc:511] Master 127.21.201.126:44537 requested a full tablet report, sending...
I20260812 00:59:03.463464 354 ts_manager.cc:194] Registered new tserver with Master: 2ac902d212d8411bb6adfa954008ed8e (127.21.201.65:37425)
I20260812 00:59:03.465085 354 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.201.65@KRBTEST.COM'} at 127.21.201.65:44351
WARNING: no policy specified for oryx/127.21.201.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.201.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.201.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.66.keytab.
Entry for principal oryx/127.21.201.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.66.keytab.
I20260812 00:59:03.503718 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:0
--local_ip_for_outbound_sockets=127.21.201.66
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:44537
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.66.keytab
--principal=oryx/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:41999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:59:03.871397 666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:03.871793 666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:03.871976 666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:03.880246 666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:03.880427 666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:03.880630 666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 00:59:03.893375 666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:41999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.66.keytab
--principal=oryx/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:44537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:03.894886 666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:03.896368 666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:03.910661 673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:03.910691 676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:03.910624 674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:03.912012 666 server_base.cc:1034] running on GCE node
I20260812 00:59:03.912744 666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:03.913883 666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:03.915166 666 hybrid_clock.cc:648] HybridClock initialized: now 1786496343915124 us; error 62 us; skew 500 ppm
I20260812 00:59:03.915545 666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:03 dist-test-slave-7f01 krb5kdc[32752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496343, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:03.920410 666 init.cc:377] Logged in from keytab as oryx/127.21.201.66@KRBTEST.COM (short username oryx)
I20260812 00:59:03.922410 666 webserver.cc:460] Webserver started at http://127.21.201.66:33705/ using document root <none> and password file <none>
I20260812 00:59:03.923225 666 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:03.923395 666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:03.923731 666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:59:03.927081 666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/data/instance:
uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828"
format_stamp: "Formatted at 2026-08-12 00:59:03 on dist-test-slave-7f01"
server_key: "791bea777970166e1504ba802cce1675"
server_key_iv: "cd55852aaef79e4c3e5256906f544859"
server_key_version: "encryptionkey@0"
I20260812 00:59:03.928085 666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance:
uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828"
format_stamp: "Formatted at 2026-08-12 00:59:03 on dist-test-slave-7f01"
server_key: "791bea777970166e1504ba802cce1675"
server_key_iv: "cd55852aaef79e4c3e5256906f544859"
server_key_version: "encryptionkey@0"
I20260812 00:59:03.934576 666 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 00:59:03.938814 683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:03.940487 666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 00:59:03.940713 666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828"
format_stamp: "Formatted at 2026-08-12 00:59:03 on dist-test-slave-7f01"
server_key: "791bea777970166e1504ba802cce1675"
server_key_iv: "cd55852aaef79e4c3e5256906f544859"
server_key_version: "encryptionkey@0"
I20260812 00:59:03.940958 666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:03.961179 666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:03.967182 666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:03.967542 666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:03.968777 666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:59:03.970520 666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:59:03.970626 666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:03.970769 666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:59:03.970813 666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:04.009454 666 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:42819
I20260812 00:59:04.009491 863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:42819 every 8 connection(s)
I20260812 00:59:04.010835 666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 00:59:04.014024 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 666
I20260812 00:59:04.014242 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance
I20260812 00:59:04.014881 22309 external_mini_cluster.cc:1428] Setting key 5331c05d535a3c443f2e90aa06e43c5f
Aug 12 00:59:04 dist-test-slave-7f01 krb5kdc[32752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496343, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM
I20260812 00:59:04.039000 866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:04.013689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:37151 (local address 127.21.201.126:44537)
0812 00:59:04.014242 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:04.014249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:04.015337 (+ 1088us) server_negotiation.cc:407] Connection header received
0812 00:59:04.017532 (+ 2195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:04.017544 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:04.017632 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:04.017741 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:04.019963 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:04.020752 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:04.023290 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:04.023825 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:04.023939 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:04.029310 (+ 5371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:04.029325 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:04.029335 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:04.029416 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:04.034481 (+ 5065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:04.035617 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:04.035700 (+ 83us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:04.035717 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:04.036031 (+ 314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:04.036583 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:04.036591 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:04.036597 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:04.037108 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:04.037277 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:04.038106 (+ 829us) server_negotiation.cc:299] Negotiation successful
0812 00:59:04.038432 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":280,"threads_started":1}
I20260812 00:59:04.039672 867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:04.014208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.21.201.66:37151)
0812 00:59:04.015139 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:04.015191 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:04.017194 (+ 2003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:04.018026 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:04.018037 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:04.018851 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:04.019764 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:04.019798 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:04.020921 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:04.020928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:04.023098 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:04.023117 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:04.024080 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:04.024090 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:04.024457 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:04.025267 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:04.025310 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:04.028722 (+ 3412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:04.034788 (+ 6066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:04.034797 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:04.034814 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:04.035420 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:04.036202 (+ 782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:04.036206 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:04.036208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:04.036427 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:04.037288 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:04.037296 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:04.037688 (+ 392us) client_negotiation.cc:769] Sending connection context
0812 00:59:04.037943 (+ 255us) client_negotiation.cc:240] Negotiation successful
0812 00:59:04.038442 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":329,"threads_started":1}
I20260812 00:59:04.041812 864 heartbeater.cc:347] Connected to a master server at 127.21.201.126:44537
I20260812 00:59:04.042169 864 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:04.043004 864 heartbeater.cc:511] Master 127.21.201.126:44537 requested a full tablet report, sending...
I20260812 00:59:04.045521 354 ts_manager.cc:194] Registered new tserver with Master: 6f446bf9ca014a99b3ddcb2f0e2bc828 (127.21.201.66:42819)
I20260812 00:59:04.046558 354 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.201.66@KRBTEST.COM'} at 127.21.201.66:37151
WARNING: no policy specified for oryx/127.21.201.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.201.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.201.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.67.keytab.
Entry for principal oryx/127.21.201.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.67.keytab.
I20260812 00:59:04.083269 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:0
--local_ip_for_outbound_sockets=127.21.201.67
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:44537
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.67.keytab
--principal=oryx/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:41999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:59:04.443584 876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:04.443982 876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:04.444092 876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:04.451902 876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:04.452245 876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:04.452491 876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 00:59:04.463346 876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:41999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.67.keytab
--principal=oryx/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:44537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:04.464759 876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:04.466415 876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:59:04.468410 659 heartbeater.cc:503] Master 127.21.201.126:44537 was elected leader, sending a full tablet report...
W20260812 00:59:04.480959 883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:04.481003 887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:04.481029 894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:04.482381 876 server_base.cc:1034] running on GCE node
I20260812 00:59:04.483487 876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:04.485013 876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:04.486304 876 hybrid_clock.cc:648] HybridClock initialized: now 1786496344486232 us; error 82 us; skew 500 ppm
I20260812 00:59:04.486701 876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:04 dist-test-slave-7f01 krb5kdc[32752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496344, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:04.491194 876 init.cc:377] Logged in from keytab as oryx/127.21.201.67@KRBTEST.COM (short username oryx)
I20260812 00:59:04.493114 876 webserver.cc:460] Webserver started at http://127.21.201.67:33361/ using document root <none> and password file <none>
I20260812 00:59:04.493880 876 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:04.494029 876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:04.494365 876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:59:04.497691 876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/data/instance:
uuid: "3a45e74843e94221ad9517a04509d375"
format_stamp: "Formatted at 2026-08-12 00:59:04 on dist-test-slave-7f01"
server_key: "d7eddd017f71f6fa27ee6b2516dadffd"
server_key_iv: "6bcde8b95b42005cdf6818f605931a35"
server_key_version: "encryptionkey@0"
I20260812 00:59:04.498592 876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance:
uuid: "3a45e74843e94221ad9517a04509d375"
format_stamp: "Formatted at 2026-08-12 00:59:04 on dist-test-slave-7f01"
server_key: "d7eddd017f71f6fa27ee6b2516dadffd"
server_key_iv: "6bcde8b95b42005cdf6818f605931a35"
server_key_version: "encryptionkey@0"
I20260812 00:59:04.504473 876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 00:59:04.508440 906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:04.510768 876 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 00:59:04.511173 876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "3a45e74843e94221ad9517a04509d375"
format_stamp: "Formatted at 2026-08-12 00:59:04 on dist-test-slave-7f01"
server_key: "d7eddd017f71f6fa27ee6b2516dadffd"
server_key_iv: "6bcde8b95b42005cdf6818f605931a35"
server_key_version: "encryptionkey@0"
I20260812 00:59:04.511554 876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:04.526273 876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:04.532033 876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:04.532452 876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:04.533739 876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:59:04.535637 876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:59:04.535725 876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:04.535959 876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:59:04.536016 876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:04.575515 876 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:36707
I20260812 00:59:04.575559 1096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:36707 every 8 connection(s)
I20260812 00:59:04.577138 876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 00:59:04.583724 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 876
I20260812 00:59:04.584020 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance
I20260812 00:59:04.584651 22309 external_mini_cluster.cc:1428] Setting key fdc7f72b555bdcd00dc4410f3cf0f5d7
Aug 12 00:59:04 dist-test-slave-7f01 krb5kdc[32752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496344, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM
I20260812 00:59:04.602752 1103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:04.580174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:54683 (local address 127.21.201.126:44537)
0812 00:59:04.580816 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:04.580821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:04.582237 (+ 1416us) server_negotiation.cc:407] Connection header received
0812 00:59:04.584404 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:04.584410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:04.584520 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:04.584649 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:04.586550 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:04.587423 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:04.589133 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:04.589565 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:04.589711 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:04.594498 (+ 4787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:04.594506 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:04.594510 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:04.594565 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:04.598039 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:04.599217 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:04.599224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:04.599226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:04.599429 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:04.600112 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:04.600119 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:04.600127 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:04.600505 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:04.600689 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:04.602052 (+ 1363us) server_negotiation.cc:299] Negotiation successful
0812 00:59:04.602334 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":378,"threads_started":1}
I20260812 00:59:04.604212 1104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:04.581151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.21.201.67:54683)
0812 00:59:04.581957 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:04.581998 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:04.584023 (+ 2025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:04.584916 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:04.584926 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:04.585611 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:04.586365 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:04.586392 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:04.587584 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:04.587589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:04.588971 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:04.588990 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:04.589816 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:04.589824 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:04.590072 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:04.590951 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:04.590984 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:04.594117 (+ 3133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:04.598287 (+ 4170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:04.598298 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:04.598319 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:04.598963 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:04.599623 (+ 660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:04.599629 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:04.599632 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:04.599953 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:04.600757 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:04.600769 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:04.601402 (+ 633us) client_negotiation.cc:769] Sending connection context
0812 00:59:04.601779 (+ 377us) client_negotiation.cc:240] Negotiation successful
0812 00:59:04.602359 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":339,"threads_started":1}
I20260812 00:59:04.606463 1098 heartbeater.cc:347] Connected to a master server at 127.21.201.126:44537
I20260812 00:59:04.606899 1098 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:04.607775 1098 heartbeater.cc:511] Master 127.21.201.126:44537 requested a full tablet report, sending...
I20260812 00:59:04.609517 354 ts_manager.cc:194] Registered new tserver with Master: 3a45e74843e94221ad9517a04509d375 (127.21.201.67:36707)
I20260812 00:59:04.610550 354 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.201.67@KRBTEST.COM'} at 127.21.201.67:54683
I20260812 00:59:04.615516 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.201.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.201.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.201.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.125.keytab.
Entry for principal oryx/127.21.201.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.125.keytab.
I20260812 00:59:04.680485 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.125:39259
--webserver_interface=127.21.201.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.125.keytab
--principal=oryx/127.21.201.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:41999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:44537,127.21.201.125:39259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:59:05.050443 864 heartbeater.cc:503] Master 127.21.201.126:44537 was elected leader, sending a full tablet report...
W20260812 00:59:05.087445 1109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:05.087908 1109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:05.088093 1109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:05.097527 1109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:59:05.097795 1109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:05.097913 1109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:05.098030 1109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:59:05.098124 1109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:59:05.111599 1109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:41999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:44537,127.21.201.125:39259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.125.keytab
--principal=oryx/127.21.201.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.125:39259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.201.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
W20260812 00:59:05.112151 1109 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.21.201.126:44537,127.21.201.125:39259'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260812 00:59:05.113332 1109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:05.114754 1109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:05.129920 1120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:05.129878 1122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:05.130651 1117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:05.131325 1109 server_base.cc:1034] running on GCE node
I20260812 00:59:05.132354 1109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:05.134064 1109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:05.135346 1109 hybrid_clock.cc:648] HybridClock initialized: now 1786496345135239 us; error 64 us; skew 500 ppm
I20260812 00:59:05.135798 1109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:05 dist-test-slave-7f01 krb5kdc[32752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496345, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:05.140406 1109 init.cc:377] Logged in from keytab as oryx/127.21.201.125@KRBTEST.COM (short username oryx)
I20260812 00:59:05.142395 1109 webserver.cc:460] Webserver started at http://127.21.201.125:33995/ using document root <none> and password file <none>
I20260812 00:59:05.143132 1109 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:05.143227 1109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:05.143509 1109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:59:05.147342 1109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-1/data/instance:
uuid: "c4c07c47632b4011bfc2a2bb1ed4a100"
format_stamp: "Formatted at 2026-08-12 00:59:05 on dist-test-slave-7f01"
server_key: "c953b71aaa2636004116ff31c364dfaf"
server_key_iv: "bd57f25f7dd821235792770a49a642a1"
server_key_version: "encryptionkey@0"
I20260812 00:59:05.148641 1109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-1/wal/instance:
uuid: "c4c07c47632b4011bfc2a2bb1ed4a100"
format_stamp: "Formatted at 2026-08-12 00:59:05 on dist-test-slave-7f01"
server_key: "c953b71aaa2636004116ff31c364dfaf"
server_key_iv: "bd57f25f7dd821235792770a49a642a1"
server_key_version: "encryptionkey@0"
I20260812 00:59:05.159523 1109 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.005s sys 0.005s
I20260812 00:59:05.164027 1135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:05.165498 1109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 00:59:05.165721 1109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-1/wal
uuid: "c4c07c47632b4011bfc2a2bb1ed4a100"
format_stamp: "Formatted at 2026-08-12 00:59:05 on dist-test-slave-7f01"
server_key: "c953b71aaa2636004116ff31c364dfaf"
server_key_iv: "bd57f25f7dd821235792770a49a642a1"
server_key_version: "encryptionkey@0"
I20260812 00:59:05.166010 1109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:05.180086 1109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:05.185920 1109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:05.186249 1109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:05.207496 1109 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.125:39259
I20260812 00:59:05.207521 1219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.125:39259 every 8 connection(s)
I20260812 00:59:05.209132 1109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-1/data/info.pb
I20260812 00:59:05.212476 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 1109
I20260812 00:59:05.213647 1220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:05.224113 1220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 00:59:05 dist-test-slave-7f01 krb5kdc[32752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.201.125@KRBTEST.COM
Aug 12 00:59:05 dist-test-slave-7f01 krb5kdc[32752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496345, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.125@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM
I20260812 00:59:05.241698 1226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:05.215340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32876 (local address 127.21.201.125:39259)
0812 00:59:05.216255 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:05.216270 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:05.216329 (+ 59us) server_negotiation.cc:407] Connection header received
0812 00:59:05.218803 (+ 2474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:05.218834 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:05.219386 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:05.219872 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:05.221448 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.223399 (+ 1951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.225318 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.226390 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.226564 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:05.232487 (+ 5923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:05.232499 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:05.232514 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:05.232573 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:05.236486 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:05.237344 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:05.237351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:05.237358 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:05.237546 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:05.238039 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:05.238045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:05.238048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:05.238813 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:05.239085 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:05.239643 (+ 558us) server_negotiation.cc:299] Negotiation successful
0812 00:59:05.240168 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"spinlock_wait_cycles":18816,"thread_start_us":417,"threads_started":1}
I20260812 00:59:05.247445 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:05.227188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.0.0.1:36684)
0812 00:59:05.228600 (+ 1412us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:05.228667 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:05.228939 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:05.229458 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:05.229467 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:05.230824 (+ 1357us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:05.231377 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:05.231401 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.232849 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.232854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:05.234389 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:05.234439 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.235542 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.235547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:05.235689 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:05.236376 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:05.236394 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:05.239378 (+ 2984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:05.243512 (+ 4134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:05.243523 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:05.243530 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:05.244270 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:05.244792 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:05.244796 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:05.244798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:05.245099 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:05.246403 (+ 1304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:05.246409 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:05.246591 (+ 182us) client_negotiation.cc:769] Sending connection context
0812 00:59:05.246785 (+ 194us) client_negotiation.cc:240] Negotiation successful
0812 00:59:05.247116 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"thread_start_us":593,"threads_started":1}
I20260812 00:59:05.247617 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:05.227474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36684 (local address 127.21.201.126:44537)
0812 00:59:05.228520 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:05.228528 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:05.228802 (+ 274us) server_negotiation.cc:407] Connection header received
0812 00:59:05.229125 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:05.229130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:05.229209 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:05.229347 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:05.231629 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.232675 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.234633 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.235334 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.235486 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:05.239598 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:05.239606 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:05.239611 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:05.239658 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:05.243339 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:05.244424 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:05.244429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:05.244432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:05.244601 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:05.245457 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:05.245467 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:05.245474 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:05.246205 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:05.246383 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:05.246981 (+ 598us) server_negotiation.cc:299] Negotiation successful
0812 00:59:05.247193 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":571,"threads_started":1}
I20260812 00:59:05.249881 1220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 00:59:05 dist-test-slave-7f01 krb5kdc[32752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496345, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.125@KRBTEST.COM for oryx/127.21.201.125@KRBTEST.COM
I20260812 00:59:05.265132 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:05.250661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:39259 (local address 127.0.0.1:32878)
0812 00:59:05.250882 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:05.250900 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:05.251013 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:05.251606 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:05.251610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:05.252052 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:05.252213 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:05.252223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.253340 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.253344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:05.254255 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:05.254266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.254968 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.254973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:05.255126 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:05.256054 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:05.256071 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:05.258827 (+ 2756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:05.262256 (+ 3429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:05.262261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:05.262264 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:05.262699 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:05.263218 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:05.263223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:05.263226 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:05.263533 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:05.264295 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:05.264300 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:05.264450 (+ 150us) client_negotiation.cc:769] Sending connection context
0812 00:59:05.264574 (+ 124us) client_negotiation.cc:240] Negotiation successful
0812 00:59:05.264835 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260812 00:59:05.265488 1226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:05.250894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32878 (local address 127.21.201.125:39259)
0812 00:59:05.251235 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:05.251241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:05.251267 (+ 26us) server_negotiation.cc:407] Connection header received
0812 00:59:05.251337 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:05.251341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:05.251422 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:05.251561 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:05.252368 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.253190 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.254398 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.254804 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.254912 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:05.259070 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:05.259075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:05.259079 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:05.259111 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:05.262089 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:05.262823 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:05.262827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:05.262829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:05.262982 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:05.263823 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:05.263827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:05.263829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:05.264142 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:05.264286 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:05.264681 (+ 395us) server_negotiation.cc:299] Negotiation successful
0812 00:59:05.265049 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":113,"server-negotiator.queue_time_us":164}
I20260812 00:59:05.271008 1220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100: Bootstrap starting.
I20260812 00:59:05.274646 1220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:05.275943 1220 log.cc:824] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100: Log is configured to *not* fsync() on all Append() calls
I20260812 00:59:05.279187 1220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100: No bootstrap required, opened a new log
I20260812 00:59:05.287204 1220 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } }
I20260812 00:59:05.287673 1220 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:05.287809 1220 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4c07c47632b4011bfc2a2bb1ed4a100, State: Initialized, Role: FOLLOWER
I20260812 00:59:05.288462 1220 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } }
I20260812 00:59:05.289505 1231 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } } }
I20260812 00:59:05.289824 1231 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:59:05.290634 1220 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 00:59:05.301926 1250 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:59:05.302217 1250 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:59:05.304874 1109 master_runner.cc:428] Detected that this master c4c07c47632b4011bfc2a2bb1ed4a100 is joining an existing cluster
I20260812 00:59:05.305022 1109 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.201.125:39259
I20260812 00:59:05.330415 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:05.311179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36698 (local address 127.21.201.126:44537)
0812 00:59:05.311422 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:05.311429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:05.312493 (+ 1064us) server_negotiation.cc:407] Connection header received
0812 00:59:05.312795 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:05.312803 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:05.312916 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:05.313012 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:05.314324 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.315070 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.317389 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.318163 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.318282 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:05.321534 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:05.321544 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:05.321551 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:05.321598 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:05.326578 (+ 4980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:05.327640 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:05.327647 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:05.327650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:05.327856 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:05.328404 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:05.328410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:05.328414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:05.328790 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:05.328902 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:05.329589 (+ 687us) server_negotiation.cc:299] Negotiation successful
0812 00:59:05.329904 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 00:59:05.331796 1264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:05.311174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.0.0.1:36698)
0812 00:59:05.312371 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:05.312396 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:05.312623 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:05.313267 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:05.313274 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:05.313853 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:05.314170 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:05.314179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.315211 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.315219 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:05.317196 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:05.317215 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.318654 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.318663 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:05.318902 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:05.319851 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:05.319879 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:05.321267 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:05.326752 (+ 5485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:05.326760 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:05.326789 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:05.327474 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:05.328029 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:05.328036 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:05.328042 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:05.328243 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:05.328981 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:05.328985 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:05.329206 (+ 221us) client_negotiation.cc:769] Sending connection context
0812 00:59:05.330734 (+ 1528us) client_negotiation.cc:240] Negotiation successful
0812 00:59:05.331165 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"thread_start_us":757,"threads_started":1}
WARNING: no policy specified for oryx/127.21.201.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.201.124@KRBTEST.COM" created.
I20260812 00:59:05.349773 1226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:05.312957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32884 (local address 127.21.201.125:39259)
0812 00:59:05.313123 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:05.313129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:05.325176 (+ 12047us) server_negotiation.cc:407] Connection header received
0812 00:59:05.325413 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:05.325418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:05.325501 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:05.325573 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:05.327188 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.328427 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.330863 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.331619 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.331728 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:05.334743 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:05.334750 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:05.334754 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:05.334796 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:05.339131 (+ 4335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:05.340603 (+ 1472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:05.340613 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:05.340620 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:05.340994 (+ 374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:05.347246 (+ 6252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:05.347254 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:05.347257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:05.347921 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:05.348131 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:05.348809 (+ 678us) server_negotiation.cc:299] Negotiation successful
0812 00:59:05.349172 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 00:59:05.353228 1266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:05.312505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:39259 (local address 127.0.0.1:32884)
0812 00:59:05.325069 (+ 12564us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:05.325090 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:05.325274 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:05.325806 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:05.325811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:05.326532 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:05.327031 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:05.327045 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.328658 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.328664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:05.330346 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:05.330367 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.332371 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.332387 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:05.332580 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:05.333307 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:05.333332 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:05.334539 (+ 1207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:05.339291 (+ 4752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:05.339296 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:05.339300 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:05.340329 (+ 1029us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:05.346745 (+ 6416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:05.346751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:05.346756 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:05.347059 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:05.348214 (+ 1155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:05.348219 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:05.348486 (+ 267us) client_negotiation.cc:769] Sending connection context
0812 00:59:05.352128 (+ 3642us) client_negotiation.cc:240] Negotiation successful
0812 00:59:05.352627 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12412,"thread_start_us":12038,"threads_started":1}
Aug 12 00:59:05 dist-test-slave-7f01 krb5kdc[32752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496342, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.126@KRBTEST.COM for oryx/127.21.201.125@KRBTEST.COM
I20260812 00:59:05.367620 1267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:05.339902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:39259 (local address 127.0.0.1:32890)
0812 00:59:05.345699 (+ 5797us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:05.345772 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:05.346077 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:05.351905 (+ 5828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:05.351920 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:05.353185 (+ 1265us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:05.353548 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:05.353565 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.355614 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.355619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:05.356780 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:05.356801 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.357632 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.357638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:05.357843 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:05.358630 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:05.358648 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:05.361092 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:05.364548 (+ 3456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:05.364564 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:05.364572 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:05.365042 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:05.365464 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:05.365469 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:05.365473 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:05.365633 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:05.366301 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:05.366310 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:05.366509 (+ 199us) client_negotiation.cc:769] Sending connection context
0812 00:59:05.366707 (+ 198us) client_negotiation.cc:240] Negotiation successful
0812 00:59:05.367200 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5421,"thread_start_us":5191,"threads_started":1}
I20260812 00:59:05.367590 1226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:05.340689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32890 (local address 127.21.201.125:39259)
0812 00:59:05.351044 (+ 10355us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:05.351053 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:05.351109 (+ 56us) server_negotiation.cc:407] Connection header received
0812 00:59:05.351274 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:05.351283 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:05.351548 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:05.351728 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:05.353802 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.355407 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.356958 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.357442 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.357588 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:05.361330 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:05.361335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:05.361338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:05.361379 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:05.364389 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:05.365192 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:05.365197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:05.365199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:05.365335 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:05.365766 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:05.365770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:05.365772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:05.366160 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:05.366323 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:05.366862 (+ 539us) server_negotiation.cc:299] Negotiation successful
0812 00:59:05.367113 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10176,"thread_start_us":12769,"threads_started":1}
I20260812 00:59:05.370369 354 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.201.125:39259: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100"
member_type: NON_VOTER
last_known_addr {
host: "127.21.201.125"
port: 39259
}
attrs {
promote: true
}
}
dest_uuid: "3ef13f6098fe4535af6f2557d272fd2b"
cas_config_opid_index: -1
I20260812 00:59:05.371942 354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: NON_VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: true } }
I20260812 00:59:05.375852 1275 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c4c07c47632b4011bfc2a2bb1ed4a100 (127.21.201.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: NON_VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: true } } }
I20260812 00:59:05.377482 1178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:05.382565 1275 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.201.125:39259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.201.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.124.keytab.
Entry for principal oryx/127.21.201.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.124.keytab.
I20260812 00:59:05.383203 1275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3ef13f6098fe4535af6f2557d272fd2b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: NON_VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: true } } }
I20260812 00:59:05.383503 1275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [sys.catalog]: This master's current role is: LEADER
I20260812 00:59:05.383687 1109 master.cc:537] Master@127.21.201.125:39259 shutting down...
I20260812 00:59:05.385355 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.124:39009
--webserver_interface=127.21.201.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.124.keytab
--principal=oryx/127.21.201.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:41999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:44537,127.21.201.125:39259,127.21.201.124:39009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:59:05.388979 1178 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 1 FOLLOWER]: Refusing update from remote peer 3ef13f6098fe4535af6f2557d272fd2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 00:59:05.390417 1275 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: NON_VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 00:59:05.394660 328 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260812 00:59:05.406050 1109 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:59:05.406566 1109 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:59:05.406807 1109 tablet_replica.cc:330] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100: stopping tablet replica
W20260812 00:59:05.406947 328 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b -> Peer c4c07c47632b4011bfc2a2bb1ed4a100 (127.21.201.125:39259): Couldn't send request to peer c4c07c47632b4011bfc2a2bb1ed4a100. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 00:59:05.414484 1109 master.cc:552] Master@127.21.201.125:39259 shutdown complete.
I20260812 00:59:05.414645 1109 master_runner.cc:305] Clearing existing system tablet
I20260812 00:59:05.415174 1109 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:05.415280 1109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:05.418625 1109 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260812 00:59:05.420589 1281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:05.420992 1109 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:05.421290 1109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260812 00:59:05.421394 1109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-1/wal
uuid: "c4c07c47632b4011bfc2a2bb1ed4a100"
format_stamp: "Formatted at 2026-08-12 00:59:05 on dist-test-slave-7f01"
server_key: "c953b71aaa2636004116ff31c364dfaf"
server_key_iv: "bd57f25f7dd821235792770a49a642a1"
server_key_version: "encryptionkey@0"
I20260812 00:59:05.427212 1109 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:59:05.434942 1109 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 00:59:05.435153 1109 log.cc:1196] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260812 00:59:05.435642 1109 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100: Deleting consensus metadata
I20260812 00:59:05.435959 1109 master_runner.cc:315] Copying system tablet from 127.21.201.126:44537
I20260812 00:59:05.439466 1109 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100: tablet copy: Beginning tablet copy session from remote peer at address 127.21.201.126:44537
I20260812 00:59:05.458810 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:05.440927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36708 (local address 127.21.201.126:44537)
0812 00:59:05.441144 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:05.441150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:05.441940 (+ 790us) server_negotiation.cc:407] Connection header received
0812 00:59:05.442148 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:05.442154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:05.442378 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:05.442525 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:05.443880 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.445107 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.447295 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.448388 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.448551 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:05.451040 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:05.451048 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:05.451053 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:05.451095 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:05.455244 (+ 4149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:05.456334 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:05.456341 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:05.456344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:05.456537 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:05.457015 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:05.457020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:05.457023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:05.457435 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:05.457606 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:05.458167 (+ 561us) server_negotiation.cc:299] Negotiation successful
0812 00:59:05.458419 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 00:59:05.458911 1299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:05.440956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.0.0.1:36708)
0812 00:59:05.441753 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:05.441797 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:05.441999 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:05.442573 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:05.442579 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:05.443205 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:05.443553 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:05.443567 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.445378 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.445384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:05.447076 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:05.447093 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.448606 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.448613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:05.448820 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:05.449582 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:05.449613 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:05.450768 (+ 1155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:05.455501 (+ 4733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:05.455507 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:05.455512 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:05.456160 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:05.456691 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:05.456697 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:05.456701 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:05.456884 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:05.457603 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:05.457610 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:05.457825 (+ 215us) client_negotiation.cc:769] Sending connection context
0812 00:59:05.458010 (+ 185us) client_negotiation.cc:240] Negotiation successful
0812 00:59:05.458404 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":482,"threads_started":1}
I20260812 00:59:05.460418 380 tablet_copy_service.cc:140] P 3ef13f6098fe4535af6f2557d272fd2b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c4c07c47632b4011bfc2a2bb1ed4a100 ({username='oryx', principal='oryx/127.21.201.125@KRBTEST.COM'} at 127.0.0.1:36708)
I20260812 00:59:05.460920 380 tablet_copy_service.cc:161] P 3ef13f6098fe4535af6f2557d272fd2b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c4c07c47632b4011bfc2a2bb1ed4a100 at {username='oryx', principal='oryx/127.21.201.125@KRBTEST.COM'} at 127.0.0.1:36708: session id = c4c07c47632b4011bfc2a2bb1ed4a100-00000000000000000000000000000000
I20260812 00:59:05.466677 380 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 00:59:05.470707 1109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 00:59:05.478950 1109 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100: tablet copy: Starting download of 0 data blocks...
I20260812 00:59:05.479473 1109 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100: tablet copy: Starting download of 1 WAL segments...
I20260812 00:59:05.483541 1109 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 00:59:05.489149 380 tablet_copy_service.cc:342] P 3ef13f6098fe4535af6f2557d272fd2b: Request end of tablet copy session c4c07c47632b4011bfc2a2bb1ed4a100-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.201.125@KRBTEST.COM'} at 127.0.0.1:36708
I20260812 00:59:05.489480 380 tablet_copy_service.cc:433] P 3ef13f6098fe4535af6f2557d272fd2b: ending tablet copy session c4c07c47632b4011bfc2a2bb1ed4a100-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c4c07c47632b4011bfc2a2bb1ed4a100
I20260812 00:59:05.504890 1109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:05.508685 1301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:05.513316 1303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:05.514429 1305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:05.515218 1109 server_base.cc:1034] running on GCE node
I20260812 00:59:05.515578 1109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:05.516435 1109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:05.517665 1109 hybrid_clock.cc:648] HybridClock initialized: now 1786496345517600 us; error 36 us; skew 500 ppm
I20260812 00:59:05.517931 1109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:05 dist-test-slave-7f01 krb5kdc[32752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496345, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:05.520059 1109 init.cc:377] Logged in from keytab as oryx/127.21.201.125@KRBTEST.COM (short username oryx)
I20260812 00:59:05.524735 1109 webserver.cc:460] Webserver started at http://127.21.201.125:35247/ using document root <none> and password file <none>
I20260812 00:59:05.525208 1109 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:05.525334 1109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:05.527899 1109 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260812 00:59:05.529804 1315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:05.530254 1109 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:59:05.530426 1109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-1/wal
uuid: "c4c07c47632b4011bfc2a2bb1ed4a100"
format_stamp: "Formatted at 2026-08-12 00:59:05 on dist-test-slave-7f01"
server_key: "c953b71aaa2636004116ff31c364dfaf"
server_key_iv: "bd57f25f7dd821235792770a49a642a1"
server_key_version: "encryptionkey@0"
I20260812 00:59:05.530612 1109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:05.555339 1109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:05.559067 1109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:05.597116 1109 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.125:39259
I20260812 00:59:05.597138 1407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.125:39259 every 8 connection(s)
I20260812 00:59:05.598361 1109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-1/data/info.pb
I20260812 00:59:05.606705 1410 sys_catalog.cc:263] Verifying existing consensus state
I20260812 00:59:05.609644 1410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100: Bootstrap starting.
I20260812 00:59:05.614095 1098 heartbeater.cc:503] Master 127.21.201.126:44537 was elected leader, sending a full tablet report...
I20260812 00:59:05.626215 1410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:59:05.626850 1410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100: Bootstrap complete.
I20260812 00:59:05.627852 1410 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: NON_VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: true } }
I20260812 00:59:05.628121 1410 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c4c07c47632b4011bfc2a2bb1ed4a100, State: Initialized, Role: LEARNER
I20260812 00:59:05.628379 1410 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: NON_VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: true } }
I20260812 00:59:05.629695 1410 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:59:05.629317 1416 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: NON_VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: true } } }
I20260812 00:59:05.629953 1416 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [sys.catalog]: This master's current role is: LEARNER
I20260812 00:59:05.640281 1440 catalog_manager.cc:1259] Loaded cluster ID: d4fe47a674c540db8f1ec527af6061d8
I20260812 00:59:05.640419 1440 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100: loading cluster ID for follower catalog manager: success
I20260812 00:59:05.642910 1440 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100: acquiring CA information for follower catalog manager: success
I20260812 00:59:05.645303 1440 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 00:59:05.857527 1279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:05.857930 1279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:05.858096 1279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:05.866158 1279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:59:05.866343 1279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:05.866446 1279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:05.866561 1279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:59:05.866642 1279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:59:05.880916 1279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:41999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:44537,127.21.201.125:39259,127.21.201.124:39009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.124.keytab
--principal=oryx/127.21.201.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.124:39009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.201.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:05.882967 1279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:05.884748 1279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:05.898454 1457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:05.898552 1448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:05.898737 1453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:05.899684 1279 server_base.cc:1034] running on GCE node
I20260812 00:59:05.900673 1279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:05.902238 1279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:05.903528 1279 hybrid_clock.cc:648] HybridClock initialized: now 1786496345903479 us; error 34 us; skew 500 ppm
I20260812 00:59:05.904068 1279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:05 dist-test-slave-7f01 krb5kdc[32752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496345, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:05.907965 1279 init.cc:377] Logged in from keytab as oryx/127.21.201.124@KRBTEST.COM (short username oryx)
I20260812 00:59:05.909723 1279 webserver.cc:460] Webserver started at http://127.21.201.124:43599/ using document root <none> and password file <none>
I20260812 00:59:05.910460 1279 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:05.910576 1279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:05.910856 1279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:59:05.914265 1279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-2/data/instance:
uuid: "2e5afbe350a9486e800c519579288778"
format_stamp: "Formatted at 2026-08-12 00:59:05 on dist-test-slave-7f01"
server_key: "d25a4d970c232b9d2b8d33fe34b56f14"
server_key_iv: "cf1fe6d9ac1794098452fa978ea6aa05"
server_key_version: "encryptionkey@0"
I20260812 00:59:05.915302 1279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-2/wal/instance:
uuid: "2e5afbe350a9486e800c519579288778"
format_stamp: "Formatted at 2026-08-12 00:59:05 on dist-test-slave-7f01"
server_key: "d25a4d970c232b9d2b8d33fe34b56f14"
server_key_iv: "cf1fe6d9ac1794098452fa978ea6aa05"
server_key_version: "encryptionkey@0"
I20260812 00:59:05.921347 1279 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 00:59:05.925158 1472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:05.927266 1279 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 00:59:05.927506 1279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-2/wal
uuid: "2e5afbe350a9486e800c519579288778"
format_stamp: "Formatted at 2026-08-12 00:59:05 on dist-test-slave-7f01"
server_key: "d25a4d970c232b9d2b8d33fe34b56f14"
server_key_iv: "cf1fe6d9ac1794098452fa978ea6aa05"
server_key_version: "encryptionkey@0"
I20260812 00:59:05.927772 1279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:05.945073 1279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:05.949980 1279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:05.950320 1279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:05.975018 1279 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.124:39009
I20260812 00:59:05.975044 1563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.124:39009 every 8 connection(s)
I20260812 00:59:05.976644 1279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-2/data/info.pb
I20260812 00:59:05.977787 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 1279
I20260812 00:59:05.982685 1564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:05.990532 1564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 00:59:05 dist-test-slave-7f01 krb5kdc[32752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.201.124@KRBTEST.COM
I20260812 00:59:06.004395 1573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:05.980022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35722 (local address 127.21.201.124:39009)
0812 00:59:05.980757 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:05.980770 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:05.980827 (+ 57us) server_negotiation.cc:407] Connection header received
0812 00:59:05.982556 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:05.982578 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:05.983013 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:05.983406 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:05.985078 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.986548 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.987990 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.988743 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.988965 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:05.993911 (+ 4946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:05.993919 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:05.993937 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:05.993987 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:05.998742 (+ 4755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:05.999777 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:05.999785 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:05.999795 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:06.000045 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:06.000514 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:06.000519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:06.000523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:06.001258 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:06.001557 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:06.002009 (+ 452us) server_negotiation.cc:299] Negotiation successful
0812 00:59:06.002486 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":322,"threads_started":1}
Aug 12 00:59:06 dist-test-slave-7f01 krb5kdc[32752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496345, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.124@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM
I20260812 00:59:06.014195 1575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:05.994309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.0.0.1:36710)
0812 00:59:05.995534 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:05.995582 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:05.995805 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:05.996387 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:05.996395 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:05.997426 (+ 1031us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:05.998125 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:05.998143 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.999490 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:05.999494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:06.001248 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:06.001260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.002312 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.002319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:06.002501 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:06.003228 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:06.003254 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:06.006648 (+ 3394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:06.010775 (+ 4127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:06.010789 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:06.010797 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:06.011304 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:06.011716 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:06.011721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:06.011725 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:06.011982 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:06.012956 (+ 974us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:06.012970 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:06.013266 (+ 296us) client_negotiation.cc:769] Sending connection context
0812 00:59:06.013532 (+ 266us) client_negotiation.cc:240] Negotiation successful
0812 00:59:06.013884 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"thread_start_us":755,"threads_started":1}
I20260812 00:59:06.015362 1574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:05.994343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36710 (local address 127.21.201.126:44537)
0812 00:59:05.995016 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:05.995026 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:05.995697 (+ 671us) server_negotiation.cc:407] Connection header received
0812 00:59:05.996014 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:05.996018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:05.996182 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:05.996302 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:05.998335 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:05.999270 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.001433 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.002079 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.002229 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:06.006909 (+ 4680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:06.006919 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:06.006926 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:06.006985 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:06.010592 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:06.011451 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:06.011461 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:06.011463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:06.011592 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:06.012141 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:06.012146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:06.012149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:06.012666 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:06.012858 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:06.014726 (+ 1868us) server_negotiation.cc:299] Negotiation successful
0812 00:59:06.014946 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":308,"threads_started":1}
I20260812 00:59:06.016944 1564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:59:06.021813 1579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:06.012135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:39259 (local address 127.0.0.1:32900)
0812 00:59:06.013815 (+ 1680us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:06.013843 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:06.014028 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:06.014997 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:06.015002 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:06.015030 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:06.015349 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:06.015365 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.017520 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.017524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:06.019360 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:06.019374 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.020808 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.020815 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:06.020993 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:06.021062 (+ 69us) client_negotiation.cc:769] Sending connection context
0812 00:59:06.021174 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 00:59:06.021326 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1497,"thread_start_us":1330,"threads_started":1}
I20260812 00:59:06.021793 1580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:06.012395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32900 (local address 127.21.201.125:39259)
0812 00:59:06.014594 (+ 2199us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:06.014601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:06.014623 (+ 22us) server_negotiation.cc:407] Connection header received
0812 00:59:06.014715 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:06.014722 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:06.014804 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:06.014914 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:06.015624 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.017313 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.019538 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.020595 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.020800 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:06.020999 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:06.021204 (+ 205us) server_negotiation.cc:299] Negotiation successful
0812 00:59:06.021349 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2059,"thread_start_us":2177,"threads_started":1}
I20260812 00:59:06.023152 1365 raft_consensus.cc:1203] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260812 00:59:06.023836 1416 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ef13f6098fe4535af6f2557d272fd2b. Latest consensus state: current_term: 1 leader_uuid: "3ef13f6098fe4535af6f2557d272fd2b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: NON_VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: true } } }
I20260812 00:59:06.024173 1416 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [sys.catalog]: This master's current role is: LEARNER
Aug 12 00:59:06 dist-test-slave-7f01 krb5kdc[32752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496345, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.124@KRBTEST.COM for oryx/127.21.201.125@KRBTEST.COM
I20260812 00:59:06.034298 1575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:06.017668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:39259 (local address 127.0.0.1:32916)
0812 00:59:06.017850 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:06.017866 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:06.017981 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:06.019033 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:06.019039 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:06.019462 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:06.019613 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:06.019622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.020934 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.020939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:06.021892 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:06.021910 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.022930 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.022938 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:06.023142 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:06.023968 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:06.024001 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:06.026342 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:06.030753 (+ 4411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:06.030759 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:06.030763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:06.031388 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:06.031882 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:06.031887 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:06.031891 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:06.032403 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:06.033433 (+ 1030us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:06.033457 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:06.033631 (+ 174us) client_negotiation.cc:769] Sending connection context
0812 00:59:06.033779 (+ 148us) client_negotiation.cc:240] Negotiation successful
0812 00:59:06.034053 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260812 00:59:06.034570 1581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:06.018021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32916 (local address 127.21.201.125:39259)
0812 00:59:06.018753 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:06.018757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:06.018772 (+ 15us) server_negotiation.cc:407] Connection header received
0812 00:59:06.018813 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:06.018816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:06.018872 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:06.019012 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:06.019850 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.020789 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.022097 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.022691 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.022789 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:06.026569 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:06.026574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:06.026577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:06.026609 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:06.030576 (+ 3967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:06.031517 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:06.031524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:06.031528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:06.031708 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:06.032630 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:06.032638 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:06.032643 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:06.033156 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:06.033467 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:06.033938 (+ 471us) server_negotiation.cc:299] Negotiation successful
0812 00:59:06.034170 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":420,"threads_started":1}
I20260812 00:59:06.035724 1564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.124" port: 39009 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 00:59:06 dist-test-slave-7f01 krb5kdc[32752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496345, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.124@KRBTEST.COM for oryx/127.21.201.124@KRBTEST.COM
I20260812 00:59:06.051614 1575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:06.036698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:39009 (local address 127.0.0.1:35726)
0812 00:59:06.036920 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:06.036942 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:06.037043 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:06.037294 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:06.037297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:06.037652 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:06.037798 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:06.037806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.038804 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.038808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:06.039958 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:06.039980 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.040844 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.040848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:06.041040 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:06.041768 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:06.041796 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:06.044725 (+ 2929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:06.048585 (+ 3860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:06.048591 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:06.048594 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:06.049317 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:06.049826 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:06.049835 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:06.049838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:06.050123 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:06.050736 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:06.050740 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:06.050879 (+ 139us) client_negotiation.cc:769] Sending connection context
0812 00:59:06.051001 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 00:59:06.051268 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260812 00:59:06.052142 1573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:06.036664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35726 (local address 127.21.201.124:39009)
0812 00:59:06.036837 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:06.036841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:06.037062 (+ 221us) server_negotiation.cc:407] Connection header received
0812 00:59:06.037118 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:06.037121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:06.037178 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:06.037260 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:06.037933 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.038682 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.040178 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.040662 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.040765 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:06.045044 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:06.045055 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:06.045067 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:06.045127 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:06.048345 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:06.049489 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:06.049494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:06.049496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:06.049669 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:06.050269 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:06.050273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:06.050276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:06.050607 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:06.050735 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:06.051194 (+ 459us) server_negotiation.cc:299] Negotiation successful
0812 00:59:06.051558 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 00:59:06.059500 1564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778: Bootstrap starting.
I20260812 00:59:06.063109 1564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:06.064352 1564 log.cc:824] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778: Log is configured to *not* fsync() on all Append() calls
I20260812 00:59:06.067868 1564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778: No bootstrap required, opened a new log
I20260812 00:59:06.076680 1564 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 39009 } }
I20260812 00:59:06.077167 1564 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:06.077418 1564 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e5afbe350a9486e800c519579288778, State: Initialized, Role: FOLLOWER
I20260812 00:59:06.078188 1564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 39009 } }
I20260812 00:59:06.079490 1583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 39009 } } }
I20260812 00:59:06.080147 1583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:59:06.080652 1564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 00:59:06.092442 1596 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:59:06.092618 1596 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:59:06.098587 1279 master_runner.cc:428] Detected that this master 2e5afbe350a9486e800c519579288778 is joining an existing cluster
I20260812 00:59:06.098744 1279 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.201.124:39009
I20260812 00:59:06.113740 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 447
I20260812 00:59:06.136050 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:06.104864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.0.0.1:36726)
0812 00:59:06.105925 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:06.105953 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:06.106228 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:06.107021 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:06.107028 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:06.108647 (+ 1619us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:06.110182 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:06.110206 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.117510 (+ 7304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.117514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:06.120718 (+ 3204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:06.120741 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.124358 (+ 3617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.124367 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:06.124638 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:06.125706 (+ 1068us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:06.125732 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:06.127279 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:06.131875 (+ 4596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:06.131879 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:06.131883 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:06.133073 (+ 1190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:06.133545 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:06.133549 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:06.133553 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:06.133814 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:06.134619 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:06.134624 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:06.134878 (+ 254us) client_negotiation.cc:769] Sending connection context
0812 00:59:06.135080 (+ 202us) client_negotiation.cc:240] Negotiation successful
0812 00:59:06.135430 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"spinlock_wait_cycles":76288,"thread_start_us":609,"threads_started":1}
I20260812 00:59:06.136600 1573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:06.105099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35740 (local address 127.21.201.124:39009)
0812 00:59:06.105288 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:06.105294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:06.116848 (+ 11554us) server_negotiation.cc:407] Connection header received
0812 00:59:06.117098 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:06.117108 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:06.117191 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:06.117306 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:06.120892 (+ 3586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.121999 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.124563 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.125594 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.125754 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:06.128544 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:06.128555 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:06.128561 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:06.128601 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:06.133359 (+ 4758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:06.134222 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:06.134227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:06.134229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:06.134369 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:06.134789 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:06.134800 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:06.134805 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:06.135168 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:06.135301 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:06.135847 (+ 546us) server_negotiation.cc:299] Negotiation successful
0812 00:59:06.136117 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 00:59:06.137055 1574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:06.104790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36726 (local address 127.21.201.126:44537)
0812 00:59:06.105072 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:06.105079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:06.106077 (+ 998us) server_negotiation.cc:407] Connection header received
0812 00:59:06.106423 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:06.106432 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:06.106573 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:06.106698 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:06.110844 (+ 4146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.117286 (+ 6442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.122776 (+ 5490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.123824 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.123999 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:06.127573 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:06.127581 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:06.127587 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:06.127633 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:06.131621 (+ 3988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:06.133227 (+ 1606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:06.133233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:06.133237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:06.133399 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:06.133966 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:06.133971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:06.133976 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:06.134382 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:06.134545 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:06.136424 (+ 1879us) server_negotiation.cc:299] Negotiation successful
0812 00:59:06.136711 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260812 00:59:06.137554 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:06.104876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:39009 (local address 127.0.0.1:35740)
0812 00:59:06.116713 (+ 11837us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:06.116730 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:06.116905 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:06.117501 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:06.117505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:06.119442 (+ 1937us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:06.119881 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:06.119901 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.122205 (+ 2304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.122213 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:06.124397 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:06.124413 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.126068 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.126081 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:06.126324 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:06.127051 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:06.127099 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:06.128329 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:06.133546 (+ 5217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:06.133550 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:06.133553 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:06.134110 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:06.134504 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:06.134509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:06.134513 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:06.134656 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:06.135377 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:06.135383 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:06.135522 (+ 139us) client_negotiation.cc:769] Sending connection context
0812 00:59:06.136809 (+ 1287us) client_negotiation.cc:240] Negotiation successful
0812 00:59:06.137119 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11726,"mutex_wait_us":219,"thread_start_us":12029,"threads_started":1}
I20260812 00:59:06.148187 1581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:06.106419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32918 (local address 127.21.201.125:39259)
0812 00:59:06.106644 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:06.106651 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:06.117811 (+ 11160us) server_negotiation.cc:407] Connection header received
0812 00:59:06.117958 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:06.117964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:06.118063 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:06.118168 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:06.120886 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.122380 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.129136 (+ 6756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.130090 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.130222 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:06.133226 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:06.133232 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:06.133236 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:06.133279 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:06.141763 (+ 8484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:06.143406 (+ 1643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:06.143417 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:06.143423 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:06.143684 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:06.145082 (+ 1398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:06.145090 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:06.145096 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:06.145740 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:06.145979 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:06.146971 (+ 992us) server_negotiation.cc:299] Negotiation successful
0812 00:59:06.147275 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Aug 12 00:59:06 dist-test-slave-7f01 krb5kdc[32752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.201.65@KRBTEST.COM
I20260812 00:59:06.149914 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:06.106049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:39259 (local address 127.0.0.1:32918)
0812 00:59:06.117601 (+ 11552us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:06.117621 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:06.117808 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:06.118202 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:06.118206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:06.120291 (+ 2085us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:06.120698 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:06.120711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.125225 (+ 4514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.125233 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:06.128857 (+ 3624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:06.128909 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.130564 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.130570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:06.130741 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:06.131447 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:06.131474 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:06.133001 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:06.141981 (+ 8980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:06.141988 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:06.142124 (+ 136us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:06.143114 (+ 990us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:06.143931 (+ 817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:06.143939 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:06.143945 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:06.144874 (+ 929us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:06.146289 (+ 1415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:06.146298 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:06.146578 (+ 280us) client_negotiation.cc:769] Sending connection context
0812 00:59:06.148768 (+ 2190us) client_negotiation.cc:240] Negotiation successful
0812 00:59:06.149216 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11391,"mutex_wait_us":3,"thread_start_us":10339,"threads_started":1}
Aug 12 00:59:06 dist-test-slave-7f01 krb5kdc[32752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496342, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.126@KRBTEST.COM for oryx/127.21.201.124@KRBTEST.COM
I20260812 00:59:06.159050 1613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:06.116129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34472 (local address 127.21.201.65:37425)
0812 00:59:06.130687 (+ 14558us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:06.130742 (+ 55us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:06.130780 (+ 38us) server_negotiation.cc:407] Connection header received
0812 00:59:06.130860 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:06.130871 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:06.131261 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:06.133510 (+ 2249us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:06.133808 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.135800 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.138559 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.139481 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.150878 (+ 11397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:06.151236 (+ 358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:06.151252 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:06.151273 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:06.151308 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:06.155192 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:06.156065 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:06.156074 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:06.156076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:06.156241 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:06.156758 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:06.156763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:06.156767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:06.157228 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:06.157569 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:06.157961 (+ 392us) server_negotiation.cc:299] Negotiation successful
0812 00:59:06.158257 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14055,"thread_start_us":14141,"threads_started":1}
I20260812 00:59:06.161041 653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:06.165107 1579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:06.147379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:39009 (local address 127.0.0.1:35744)
0812 00:59:06.147608 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:06.147632 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:06.147787 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:06.148443 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:06.148449 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:06.149066 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:06.149285 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:06.149298 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.151282 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.151289 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:06.152814 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:06.152834 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.153890 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.153896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:06.154119 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:06.154747 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:06.154772 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:06.157569 (+ 2797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:06.162013 (+ 4444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:06.162018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:06.162023 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:06.162451 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:06.162920 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:06.162927 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:06.162933 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:06.163122 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:06.163932 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:06.163941 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:06.164147 (+ 206us) client_negotiation.cc:769] Sending connection context
0812 00:59:06.164293 (+ 146us) client_negotiation.cc:240] Negotiation successful
0812 00:59:06.164652 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 00:59:06.166393 1573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:06.147810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35744 (local address 127.21.201.124:39009)
0812 00:59:06.147990 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:06.147996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:06.148015 (+ 19us) server_negotiation.cc:407] Connection header received
0812 00:59:06.148116 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:06.148121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:06.148231 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:06.149713 (+ 1482us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:06.149908 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.151097 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.152999 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.153714 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.153868 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:06.157785 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:06.157795 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:06.157799 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:06.157839 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:06.161809 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:06.162590 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:06.162597 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:06.162602 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:06.162785 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:06.163267 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:06.163275 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:06.163281 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:06.163718 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:06.165500 (+ 1782us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:06.165628 (+ 128us) server_negotiation.cc:299] Negotiation successful
0812 00:59:06.165918 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 00:59:06.168145 354 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.201.124:39009: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2e5afbe350a9486e800c519579288778"
member_type: NON_VOTER
last_known_addr {
host: "127.21.201.124"
port: 39009
}
attrs {
promote: true
}
}
dest_uuid: "3ef13f6098fe4535af6f2557d272fd2b"
cas_config_opid_index: 5
I20260812 00:59:06.169332 354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: NON_VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: true } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: NON_VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: true } }
I20260812 00:59:06.171224 1577 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2e5afbe350a9486e800c519579288778 (127.21.201.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: NON_VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: true } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: NON_VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: true } } }
I20260812 00:59:06.184432 1522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:06.188064 1577 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.201.124:39009
I20260812 00:59:06.188788 1522 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 1 FOLLOWER]: Refusing update from remote peer 3ef13f6098fe4535af6f2557d272fd2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 00:59:06.188881 1279 master.cc:537] Master@127.21.201.124:39009 shutting down...
I20260812 00:59:06.192101 1583 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:59:06.192420 1583 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 39009 } }
I20260812 00:59:06.194309 1583 leader_election.cc:288] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ef13f6098fe4535af6f2557d272fd2b (127.21.201.126:44537), c4c07c47632b4011bfc2a2bb1ed4a100 (127.21.201.125:39259)
I20260812 00:59:06.195014 367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e5afbe350a9486e800c519579288778" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ef13f6098fe4535af6f2557d272fd2b" is_pre_election: true
I20260812 00:59:06.195206 1365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e5afbe350a9486e800c519579288778" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" is_pre_election: true
I20260812 00:59:06.196808 1478 leader_election.cc:302] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e5afbe350a9486e800c519579288778; no voters: 3ef13f6098fe4535af6f2557d272fd2b, c4c07c47632b4011bfc2a2bb1ed4a100
I20260812 00:59:06.197973 1583 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 00:59:06.201110 1619 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: NON_VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 00:59:06.203080 1365 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 1 LEARNER]: Refusing update from remote peer 3ef13f6098fe4535af6f2557d272fd2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 00:59:06.206487 1616 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: NON_VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 00:59:06.209535 1365 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2e5afbe350a9486e800c519579288778 (127.21.201.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: NON_VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: true } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: NON_VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: true } } }
I20260812 00:59:06.211540 1618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3ef13f6098fe4535af6f2557d272fd2b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: NON_VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: true } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: NON_VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: true } } }
I20260812 00:59:06.211982 1618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [sys.catalog]: This master's current role is: LEADER
W20260812 00:59:06.214716 327 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b -> Peer 2e5afbe350a9486e800c519579288778 (127.21.201.124:39009): Couldn't send request to peer 2e5afbe350a9486e800c519579288778. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 00:59:06.215238 1416 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ef13f6098fe4535af6f2557d272fd2b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: NON_VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: true } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: NON_VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: true } } }
I20260812 00:59:06.215526 1416 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [sys.catalog]: This master's current role is: LEARNER
W20260812 00:59:06.216143 1484 tablet.cc:2369] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:59:06.216537 1577 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b: attempting to promote NON_VOTER c4c07c47632b4011bfc2a2bb1ed4a100 to VOTER
I20260812 00:59:06.218322 1279 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:59:06.218739 1279 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:59:06.218950 1279 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778: stopping tablet replica
I20260812 00:59:06.217388 1577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: NON_VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: true } }
I20260812 00:59:06.228223 1364 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 1 LEARNER]: Refusing update from remote peer 3ef13f6098fe4535af6f2557d272fd2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260812 00:59:06.229003 1577 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 00:59:06.231356 1279 master.cc:552] Master@127.21.201.124:39009 shutdown complete.
W20260812 00:59:06.231365 327 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b -> Peer 2e5afbe350a9486e800c519579288778 (127.21.201.124:39009): Couldn't send request to peer 2e5afbe350a9486e800c519579288778. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260812 00:59:06.231550 1279 master_runner.cc:305] Clearing existing system tablet
I20260812 00:59:06.232098 1279 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:06.232312 1279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:06.232888 1624 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c4c07c47632b4011bfc2a2bb1ed4a100 (127.21.201.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: NON_VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: true } } }
I20260812 00:59:06.237824 1619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3ef13f6098fe4535af6f2557d272fd2b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: NON_VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: true } } }
I20260812 00:59:06.238147 1619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [sys.catalog]: This master's current role is: LEADER
I20260812 00:59:06.239230 1364 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c4c07c47632b4011bfc2a2bb1ed4a100 (127.21.201.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: NON_VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: true } } }
I20260812 00:59:06.242255 1279 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.005s sys 0.001s
I20260812 00:59:06.245709 1616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ef13f6098fe4535af6f2557d272fd2b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: NON_VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: true } } }
I20260812 00:59:06.246234 1616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [sys.catalog]: This master's current role is: LEADER
I20260812 00:59:06.246836 1416 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ef13f6098fe4535af6f2557d272fd2b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: NON_VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: true } } }
I20260812 00:59:06.247186 1416 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:59:06.254117 1629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:06.254808 1279 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:06.255393 1279 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.003s sys 0.007s
I20260812 00:59:06.255582 1279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-2/wal
uuid: "2e5afbe350a9486e800c519579288778"
format_stamp: "Formatted at 2026-08-12 00:59:05 on dist-test-slave-7f01"
server_key: "d25a4d970c232b9d2b8d33fe34b56f14"
server_key_iv: "cf1fe6d9ac1794098452fa978ea6aa05"
server_key_version: "encryptionkey@0"
I20260812 00:59:06.262184 1279 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:59:06.269867 1279 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 00:59:06.270084 1279 log.cc:1196] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260812 00:59:06.270612 1279 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778: Deleting consensus metadata
I20260812 00:59:06.270896 1279 master_runner.cc:315] Copying system tablet from 127.21.201.126:44537
I20260812 00:59:06.287926 1279 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778: tablet copy: Beginning tablet copy session from remote peer at address 127.21.201.126:44537
I20260812 00:59:06.306629 1574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:06.289351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36740 (local address 127.21.201.126:44537)
0812 00:59:06.289610 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:06.289617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:06.289646 (+ 29us) server_negotiation.cc:407] Connection header received
0812 00:59:06.289923 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:06.289929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:06.290021 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:06.290141 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:06.291351 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.292565 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.294532 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.295220 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.295314 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:06.298700 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:06.298708 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:06.298713 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:06.298756 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:06.303206 (+ 4450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:06.304241 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:06.304248 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:06.304252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:06.304438 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:06.304921 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:06.304926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:06.304930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:06.305285 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:06.305424 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:06.306011 (+ 587us) server_negotiation.cc:299] Negotiation successful
0812 00:59:06.306234 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260812 00:59:06.308074 1635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:06.288857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.0.0.1:36740)
0812 00:59:06.289487 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:06.289515 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:06.289669 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:06.290193 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:06.290198 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:06.290847 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:06.291165 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:06.291178 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.292781 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.292787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:06.294337 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:06.294353 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.295625 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.295630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:06.296275 (+ 645us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:06.297120 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:06.297143 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:06.298432 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:06.303419 (+ 4987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:06.303425 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:06.303429 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:06.304090 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:06.304598 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:06.304603 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:06.304606 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:06.304784 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:06.305520 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:06.305526 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:06.305730 (+ 204us) client_negotiation.cc:769] Sending connection context
0812 00:59:06.307187 (+ 1457us) client_negotiation.cc:240] Negotiation successful
0812 00:59:06.307581 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":795,"threads_started":1}
I20260812 00:59:06.309062 380 tablet_copy_service.cc:140] P 3ef13f6098fe4535af6f2557d272fd2b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2e5afbe350a9486e800c519579288778 ({username='oryx', principal='oryx/127.21.201.124@KRBTEST.COM'} at 127.0.0.1:36740)
I20260812 00:59:06.309248 380 tablet_copy_service.cc:161] P 3ef13f6098fe4535af6f2557d272fd2b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2e5afbe350a9486e800c519579288778 at {username='oryx', principal='oryx/127.21.201.124@KRBTEST.COM'} at 127.0.0.1:36740: session id = 2e5afbe350a9486e800c519579288778-00000000000000000000000000000000
I20260812 00:59:06.312739 380 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 00:59:06.314634 1279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 00:59:06.321131 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 447
I20260812 00:59:06.321457 1279 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778: tablet copy: Starting download of 0 data blocks...
I20260812 00:59:06.328069 1279 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778: tablet copy: Starting download of 1 WAL segments...
I20260812 00:59:06.330976 1279 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 00:59:06.335970 380 tablet_copy_service.cc:342] P 3ef13f6098fe4535af6f2557d272fd2b: Request end of tablet copy session 2e5afbe350a9486e800c519579288778-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.201.124@KRBTEST.COM'} at 127.0.0.1:36740
I20260812 00:59:06.336283 380 tablet_copy_service.cc:433] P 3ef13f6098fe4535af6f2557d272fd2b: ending tablet copy session 2e5afbe350a9486e800c519579288778-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2e5afbe350a9486e800c519579288778
I20260812 00:59:06.357587 1279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:06.360723 1637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:06.366526 1641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:06.369439 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:37425
--local_ip_for_outbound_sockets=127.21.201.65
--tserver_master_addrs=127.21.201.126:44537
--webserver_port=34085
--webserver_interface=127.21.201.65
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.65.keytab
--principal=oryx/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:41999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.201.126:44537,127.21.201.125:39259
--tserver_master_addrs=127.21.201.126:44537,127.21.201.125:39259,127.21.201.124:39009 with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:59:06.372771 1279 server_base.cc:1034] running on GCE node
W20260812 00:59:06.373147 1647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:06.373652 1279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:06.374434 1279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:06.375663 1279 hybrid_clock.cc:648] HybridClock initialized: now 1786496346375616 us; error 34 us; skew 500 ppm
I20260812 00:59:06.376021 1279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:06 dist-test-slave-7f01 krb5kdc[32752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496346, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:06.378211 1279 init.cc:377] Logged in from keytab as oryx/127.21.201.124@KRBTEST.COM (short username oryx)
I20260812 00:59:06.380066 1279 webserver.cc:460] Webserver started at http://127.21.201.124:41617/ using document root <none> and password file <none>
I20260812 00:59:06.380522 1279 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:06.380643 1279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:06.384245 1279 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260812 00:59:06.386588 1664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:06.387110 1279 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260812 00:59:06.387250 1279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-2/wal
uuid: "2e5afbe350a9486e800c519579288778"
format_stamp: "Formatted at 2026-08-12 00:59:05 on dist-test-slave-7f01"
server_key: "d25a4d970c232b9d2b8d33fe34b56f14"
server_key_iv: "cf1fe6d9ac1794098452fa978ea6aa05"
server_key_version: "encryptionkey@0"
I20260812 00:59:06.387473 1279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:06.405486 1279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:06.409716 1279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:06.447443 1279 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.124:39009
I20260812 00:59:06.447511 1745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.124:39009 every 8 connection(s)
I20260812 00:59:06.448506 1279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/master-2/data/info.pb
I20260812 00:59:06.456274 1746 sys_catalog.cc:263] Verifying existing consensus state
I20260812 00:59:06.461952 1746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778: Bootstrap starting.
I20260812 00:59:06.478206 1746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:59:06.478758 1746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778: Bootstrap complete.
I20260812 00:59:06.479722 1746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: NON_VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: true } }
I20260812 00:59:06.480057 1746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2e5afbe350a9486e800c519579288778, State: Initialized, Role: LEARNER
I20260812 00:59:06.480337 1746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: NON_VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: true } }
I20260812 00:59:06.481186 1752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: NON_VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: true } } }
I20260812 00:59:06.481549 1746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:59:06.481700 1752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [sys.catalog]: This master's current role is: LEARNER
I20260812 00:59:06.491384 1769 catalog_manager.cc:1259] Loaded cluster ID: d4fe47a674c540db8f1ec527af6061d8
I20260812 00:59:06.491626 1769 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778: loading cluster ID for follower catalog manager: success
I20260812 00:59:06.494114 1769 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778: acquiring CA information for follower catalog manager: success
I20260812 00:59:06.496162 1769 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 00:59:06.651021 1771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:06.643697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35748 (local address 127.21.201.124:39009)
0812 00:59:06.644831 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:06.644836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:06.644856 (+ 20us) server_negotiation.cc:407] Connection header received
0812 00:59:06.644925 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:06.644928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:06.645057 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:06.645197 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:06.645791 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.647052 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.648658 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.649827 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.650060 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:06.650283 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:06.650460 (+ 177us) server_negotiation.cc:299] Negotiation successful
0812 00:59:06.650617 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1012,"thread_start_us":813,"threads_started":1}
I20260812 00:59:06.651005 1579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:06.643494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:39009 (local address 127.0.0.1:35748)
0812 00:59:06.643820 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:06.643844 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:06.644016 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:06.645272 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:06.645277 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:06.645305 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:06.645553 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:06.645566 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.647225 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.647231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:06.648500 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:06.648512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:06.650027 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:06.650036 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:06.650268 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:06.650306 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 00:59:06.650433 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 00:59:06.650603 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260812 00:59:06.652431 1703 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260812 00:59:06.653050 1752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ef13f6098fe4535af6f2557d272fd2b. Latest consensus state: current_term: 1 leader_uuid: "3ef13f6098fe4535af6f2557d272fd2b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: NON_VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: true } } }
I20260812 00:59:06.653395 1752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [sys.catalog]: This master's current role is: LEARNER
W20260812 00:59:06.857743 1657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:06.858083 1657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:06.858191 1657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:06.865907 1657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:06.866055 1657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:06.866178 1657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 00:59:06.876905 1657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:41999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.65.keytab
--principal=oryx/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:37425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=34085
--tserver_master_addrs=127.21.201.126:44537,127.21.201.125:39259,127.21.201.124:39009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:06.878400 1657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:06.879849 1657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:06.895349 1783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:06.895488 1779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:06.895401 1778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:06.897540 1657 server_base.cc:1034] running on GCE node
I20260812 00:59:06.898382 1657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:06.899681 1657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:06.900969 1657 hybrid_clock.cc:648] HybridClock initialized: now 1786496346900897 us; error 57 us; skew 500 ppm
I20260812 00:59:06.901360 1657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:06 dist-test-slave-7f01 krb5kdc[32752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496346, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:06.905278 1657 init.cc:377] Logged in from keytab as oryx/127.21.201.65@KRBTEST.COM (short username oryx)
I20260812 00:59:06.907280 1657 webserver.cc:460] Webserver started at http://127.21.201.65:34085/ using document root <none> and password file <none>
I20260812 00:59:06.908167 1657 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:06.908342 1657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:06.914556 1657 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 00:59:06.918761 1796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:06.920487 1657 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 00:59:06.920688 1657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "2ac902d212d8411bb6adfa954008ed8e"
format_stamp: "Formatted at 2026-08-12 00:59:03 on dist-test-slave-7f01"
server_key: "5187f392bdf1b733b59334ebacb840b3"
server_key_iv: "1f9d72d4a279f22041c4922844e8d153"
server_key_version: "encryptionkey@0"
I20260812 00:59:06.921803 1657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:06.943424 1657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:06.948391 1657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:06.948719 1657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:06.949895 1657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:59:06.951536 1657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:59:06.951670 1657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:06.951833 1657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:59:06.951932 1657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:07.000653 1657 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:37425
I20260812 00:59:07.000756 1984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:37425 every 8 connection(s)
I20260812 00:59:07.002481 1657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 00:59:07.007793 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 1657
I20260812 00:59:07.008191 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 666
Aug 12 00:59:07 dist-test-slave-7f01 krb5kdc[32752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496346, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.124@KRBTEST.COM
Aug 12 00:59:07 dist-test-slave-7f01 krb5kdc[32752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.201.66@KRBTEST.COM
Aug 12 00:59:07 dist-test-slave-7f01 krb5kdc[32752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496346, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM
I20260812 00:59:07.036855 1771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:07.005636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:38203 (local address 127.21.201.124:39009)
0812 00:59:07.005904 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:07.005910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:07.007830 (+ 1920us) server_negotiation.cc:407] Connection header received
0812 00:59:07.011900 (+ 4070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:07.011906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:07.012002 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:07.012091 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:07.015019 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.016557 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.020158 (+ 3601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.020861 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.020987 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:07.027435 (+ 6448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:07.027441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:07.027445 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:07.027495 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:07.032048 (+ 4553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:07.033396 (+ 1348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:07.033401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:07.033405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:07.033635 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:07.034194 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:07.034199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:07.034202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:07.034595 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:07.034741 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:07.035953 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0812 00:59:07.036247 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260812 00:59:07.040343 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:07.006314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:39009 (local address 127.21.201.65:38203)
0812 00:59:07.007582 (+ 1268us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:07.007644 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:07.011185 (+ 3541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:07.012427 (+ 1242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:07.012439 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:07.013493 (+ 1054us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:07.014699 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:07.014741 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.016889 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.016897 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:07.019945 (+ 3048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:07.019970 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.021458 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.021464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:07.021771 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:07.022681 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:07.022734 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:07.027110 (+ 4376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:07.032254 (+ 5144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:07.032266 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:07.032311 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:07.033217 (+ 906us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:07.033786 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:07.033790 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:07.033794 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:07.034061 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:07.034871 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:07.034880 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:07.035443 (+ 563us) client_negotiation.cc:769] Sending connection context
0812 00:59:07.037538 (+ 2095us) client_negotiation.cc:240] Negotiation successful
0812 00:59:07.038358 (+ 820us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"thread_start_us":614,"threads_started":1}
I20260812 00:59:07.040980 1989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:07.005631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:48309 (local address 127.21.201.126:44537)
0812 00:59:07.006821 (+ 1190us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:07.006828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:07.010116 (+ 3288us) server_negotiation.cc:407] Connection header received
0812 00:59:07.011532 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:07.011539 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:07.011631 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:07.012277 (+ 646us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:07.021278 (+ 9001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.022328 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.026758 (+ 4430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.027461 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.027570 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:07.033811 (+ 6241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:07.033821 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:07.033828 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:07.033885 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:07.037554 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:07.038612 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:07.038618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:07.038621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:07.038766 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:07.039238 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:07.039243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:07.039246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:07.039624 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:07.039770 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:07.040335 (+ 565us) server_negotiation.cc:299] Negotiation successful
0812 00:59:07.040595 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1019,"thread_start_us":903,"threads_started":1}
I20260812 00:59:07.041934 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:07.009431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38214 (local address 127.21.201.66:42819)
0812 00:59:07.010186 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:07.010195 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:07.012015 (+ 1820us) server_negotiation.cc:407] Connection header received
0812 00:59:07.012103 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:07.012112 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:07.012405 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:07.012580 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:07.015851 (+ 3271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.018113 (+ 2262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.020759 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.021991 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.022152 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:07.031927 (+ 9775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:07.031952 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:07.031978 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:07.032053 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:07.036956 (+ 4903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:07.038219 (+ 1263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:07.038230 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:07.038236 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:07.038427 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:07.038920 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:07.038925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:07.038928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:07.039408 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:07.039720 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:07.041072 (+ 1352us) server_negotiation.cc:299] Negotiation successful
0812 00:59:07.041426 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":2927,"threads_started":1}
I20260812 00:59:07.042217 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:07.006323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.21.201.65:48309)
0812 00:59:07.009990 (+ 3667us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:07.010017 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:07.011185 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:07.012553 (+ 1368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:07.012559 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:07.020388 (+ 7829us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:07.020723 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:07.020735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.022585 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.022593 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:07.024462 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:07.024480 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.027933 (+ 3453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.027939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:07.028133 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:07.029602 (+ 1469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:07.029631 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:07.033003 (+ 3372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:07.037726 (+ 4723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:07.037730 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:07.037734 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:07.038471 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:07.038939 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:07.038943 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:07.038946 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:07.039093 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:07.039878 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:07.039882 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:07.040045 (+ 163us) client_negotiation.cc:769] Sending connection context
0812 00:59:07.041244 (+ 1199us) client_negotiation.cc:240] Negotiation successful
0812 00:59:07.041646 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3507,"mutex_wait_us":74,"spinlock_wait_cycles":68992,"thread_start_us":2173,"threads_started":1}
I20260812 00:59:07.043144 1985 heartbeater.cc:347] Connected to a master server at 127.21.201.124:39009
Aug 12 00:59:07 dist-test-slave-7f01 krb5kdc[32752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496346, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.125@KRBTEST.COM
I20260812 00:59:07.043591 1985 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:07.044072 856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:07.044699 1985 heartbeater.cc:511] Master 127.21.201.124:39009 requested a full tablet report, sending...
I20260812 00:59:07.045665 1986 heartbeater.cc:347] Connected to a master server at 127.21.201.126:44537
I20260812 00:59:07.045812 1986 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:07.046357 1986 heartbeater.cc:511] Master 127.21.201.126:44537 requested a full tablet report, sending...
I20260812 00:59:07.047977 354 ts_manager.cc:194] Re-registered known tserver with Master: 2ac902d212d8411bb6adfa954008ed8e (127.21.201.65:37425)
I20260812 00:59:07.048545 1687 ts_manager.cc:194] Registered new tserver with Master: 2ac902d212d8411bb6adfa954008ed8e (127.21.201.65:37425)
I20260812 00:59:07.049517 354 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.201.65@KRBTEST.COM'} at 127.21.201.65:48309
I20260812 00:59:07.053237 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:07.012111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:55301 (local address 127.21.201.125:39259)
0812 00:59:07.029338 (+ 17227us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:07.029345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:07.033903 (+ 4558us) server_negotiation.cc:407] Connection header received
0812 00:59:07.033978 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:07.033983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:07.034091 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:07.034201 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:07.035679 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.037028 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.039021 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.039677 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.039817 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:07.045330 (+ 5513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:07.045341 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:07.045349 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:07.045409 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:07.050025 (+ 4616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:07.050983 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:07.050992 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:07.050997 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:07.051158 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:07.051698 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:07.051705 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:07.051711 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:07.052101 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:07.052232 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:07.052736 (+ 504us) server_negotiation.cc:299] Negotiation successful
0812 00:59:07.052940 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17066,"thread_start_us":17235,"threads_started":1}
I20260812 00:59:07.053476 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:07.011949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:39259 (local address 127.21.201.65:55301)
0812 00:59:07.033610 (+ 21661us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:07.033630 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:07.033767 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:07.034416 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:07.034420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:07.035050 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:07.035493 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:07.035506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.037223 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.037228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:07.038835 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:07.038849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.040204 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.040214 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:07.040448 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:07.041102 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:07.041132 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:07.044770 (+ 3638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:07.050154 (+ 5384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:07.050162 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:07.050168 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:07.050811 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:07.051313 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:07.051320 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:07.051327 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:07.051549 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:07.052318 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:07.052323 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:07.052479 (+ 156us) client_negotiation.cc:769] Sending connection context
0812 00:59:07.052629 (+ 150us) client_negotiation.cc:240] Negotiation successful
0812 00:59:07.052985 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21505,"mutex_wait_us":1,"thread_start_us":21320,"threads_started":1}
I20260812 00:59:07.054977 1987 heartbeater.cc:347] Connected to a master server at 127.21.201.125:39259
I20260812 00:59:07.055174 1987 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:07.055624 1987 heartbeater.cc:511] Master 127.21.201.125:39259 requested a full tablet report, sending...
I20260812 00:59:07.057564 1351 ts_manager.cc:194] Registered new tserver with Master: 2ac902d212d8411bb6adfa954008ed8e (127.21.201.65:37425)
I20260812 00:59:07.122496 1618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ef13f6098fe4535af6f2557d272fd2b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: NON_VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: true } } }
I20260812 00:59:07.123123 1618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [sys.catalog]: This master's current role is: LEADER
I20260812 00:59:07.124590 2001 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b: attempting to promote NON_VOTER 2e5afbe350a9486e800c519579288778 to VOTER
I20260812 00:59:07.125254 2001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: false } }
I20260812 00:59:07.127458 1364 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 1 FOLLOWER]: Refusing update from remote peer 3ef13f6098fe4535af6f2557d272fd2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260812 00:59:07.127493 1703 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 1 LEARNER]: Refusing update from remote peer 3ef13f6098fe4535af6f2557d272fd2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260812 00:59:07.128156 1618 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260812 00:59:07.128544 2001 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260812 00:59:07.131539 1618 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2e5afbe350a9486e800c519579288778 (127.21.201.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: false } } }
I20260812 00:59:07.131651 1364 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2e5afbe350a9486e800c519579288778 (127.21.201.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: false } } }
I20260812 00:59:07.136091 1618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3ef13f6098fe4535af6f2557d272fd2b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: false } } }
I20260812 00:59:07.136332 1618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [sys.catalog]: This master's current role is: LEADER
I20260812 00:59:07.136538 1618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ef13f6098fe4535af6f2557d272fd2b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: false } } }
I20260812 00:59:07.136723 1618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [sys.catalog]: This master's current role is: LEADER
I20260812 00:59:07.136902 1618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ef13f6098fe4535af6f2557d272fd2b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: false } } }
I20260812 00:59:07.137068 1618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b [sys.catalog]: This master's current role is: LEADER
I20260812 00:59:07.137041 1703 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2e5afbe350a9486e800c519579288778 (127.21.201.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: false } } }
I20260812 00:59:07.142735 1752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ef13f6098fe4535af6f2557d272fd2b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: false } } }
I20260812 00:59:07.143151 1752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:59:07.143013 2008 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ef13f6098fe4535af6f2557d272fd2b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: false } } }
I20260812 00:59:07.143489 2008 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:59:07.149912 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 666
I20260812 00:59:07.190917 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:42819
--local_ip_for_outbound_sockets=127.21.201.66
--tserver_master_addrs=127.21.201.126:44537
--webserver_port=33705
--webserver_interface=127.21.201.66
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.66.keytab
--principal=oryx/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:41999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.201.126:44537,127.21.201.125:39259
--tserver_master_addrs=127.21.201.126:44537,127.21.201.125:39259,127.21.201.124:39009 with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:59:07.557670 2009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:07.557996 2009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:07.558284 2009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:07.568233 2009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:07.568392 2009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:07.568529 2009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 00:59:07.581198 2009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:41999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.66.keytab
--principal=oryx/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:42819
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=33705
--tserver_master_addrs=127.21.201.126:44537,127.21.201.125:39259,127.21.201.124:39009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:07.582974 2009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:07.584841 2009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:07.600163 2023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:07.600421 2028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:07.600176 2021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:07.601675 2009 server_base.cc:1034] running on GCE node
I20260812 00:59:07.602439 2009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:07.603667 2009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:07.605024 2009 hybrid_clock.cc:648] HybridClock initialized: now 1786496347604881 us; error 137 us; skew 500 ppm
I20260812 00:59:07.605435 2009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:07 dist-test-slave-7f01 krb5kdc[32752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496347, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:07.609730 2009 init.cc:377] Logged in from keytab as oryx/127.21.201.66@KRBTEST.COM (short username oryx)
I20260812 00:59:07.611408 2009 webserver.cc:460] Webserver started at http://127.21.201.66:33705/ using document root <none> and password file <none>
I20260812 00:59:07.612324 2009 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:07.612509 2009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:07.618825 2009 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 00:59:07.621973 2035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:07.623449 2009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 00:59:07.623627 2009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828"
format_stamp: "Formatted at 2026-08-12 00:59:03 on dist-test-slave-7f01"
server_key: "791bea777970166e1504ba802cce1675"
server_key_iv: "cd55852aaef79e4c3e5256906f544859"
server_key_version: "encryptionkey@0"
I20260812 00:59:07.624765 2009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:07.654393 2009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:07.659559 2009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:07.659978 2009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:07.661123 2009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:59:07.662794 2009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:59:07.662894 2009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:07.663046 2009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:59:07.663156 2009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:07.701669 2009 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:42819
I20260812 00:59:07.701768 2225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:42819 every 8 connection(s)
I20260812 00:59:07.703220 2009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 00:59:07.710387 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 2009
I20260812 00:59:07.710721 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 876
Aug 12 00:59:07 dist-test-slave-7f01 krb5kdc[32752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496347, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM
Aug 12 00:59:07 dist-test-slave-7f01 krb5kdc[32752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496347, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.125@KRBTEST.COM
Aug 12 00:59:07 dist-test-slave-7f01 krb5kdc[32752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496347, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.124@KRBTEST.COM
Aug 12 00:59:07 dist-test-slave-7f01 krb5kdc[32752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.201.67@KRBTEST.COM
I20260812 00:59:07.743395 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:07.706905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:56981 (local address 127.21.201.126:44537)
0812 00:59:07.707434 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:07.707440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:07.708551 (+ 1111us) server_negotiation.cc:407] Connection header received
0812 00:59:07.711171 (+ 2620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:07.711178 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:07.711279 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:07.711390 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:07.714906 (+ 3516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.716534 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.720455 (+ 3921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.721406 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.721540 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:07.730688 (+ 9148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:07.730695 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:07.730700 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:07.730752 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:07.735406 (+ 4654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:07.738293 (+ 2887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:07.738300 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:07.738305 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:07.738520 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:07.739877 (+ 1357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:07.739884 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:07.739889 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:07.740308 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:07.740468 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:07.742662 (+ 2194us) server_negotiation.cc:299] Negotiation successful
0812 00:59:07.742956 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":235,"threads_started":1}
I20260812 00:59:07.743511 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:07.707678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.21.201.66:56981)
0812 00:59:07.708439 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:07.708463 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:07.710856 (+ 2393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:07.713005 (+ 2149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:07.713012 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:07.713549 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:07.714550 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:07.714585 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.716903 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.716912 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:07.720219 (+ 3307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:07.720241 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.722187 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.722196 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:07.722659 (+ 463us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:07.724908 (+ 2249us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:07.724940 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:07.730433 (+ 5493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:07.737382 (+ 6949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:07.737394 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:07.737415 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:07.738074 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:07.738891 (+ 817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:07.738896 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:07.738900 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:07.739141 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:07.740562 (+ 1421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:07.740571 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:07.741064 (+ 493us) client_negotiation.cc:769] Sending connection context
0812 00:59:07.741389 (+ 325us) client_negotiation.cc:240] Negotiation successful
0812 00:59:07.741979 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":13,"spinlock_wait_cycles":12928,"thread_start_us":902,"threads_started":1}
I20260812 00:59:07.744452 2235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:07.706533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:46297 (local address 127.21.201.124:39009)
0812 00:59:07.707546 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:07.707555 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:07.708389 (+ 834us) server_negotiation.cc:407] Connection header received
0812 00:59:07.711675 (+ 3286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:07.711681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:07.711812 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:07.711939 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:07.717559 (+ 5620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.719111 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.723339 (+ 4228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.724332 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.724468 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:07.733058 (+ 8590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:07.733065 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:07.733070 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:07.733115 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:07.736746 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:07.739300 (+ 2554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:07.739306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:07.739311 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:07.739503 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:07.739970 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:07.739975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:07.739979 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:07.740337 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:07.740480 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:07.743669 (+ 3189us) server_negotiation.cc:299] Negotiation successful
0812 00:59:07.744015 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":875,"thread_start_us":760,"threads_started":1}
I20260812 00:59:07.745211 2230 heartbeater.cc:347] Connected to a master server at 127.21.201.126:44537
I20260812 00:59:07.745647 2230 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:07.745769 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:07.707152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:39009 (local address 127.21.201.66:46297)
0812 00:59:07.708200 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:07.708261 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:07.710857 (+ 2596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:07.712261 (+ 1404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:07.712268 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:07.714262 (+ 1994us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:07.717226 (+ 2964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:07.717245 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.719318 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.719323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:07.721540 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:07.721560 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.726566 (+ 5006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.726572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:07.726790 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:07.728071 (+ 1281us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:07.728092 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:07.732790 (+ 4698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:07.738205 (+ 5415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:07.738210 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:07.738214 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:07.738763 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:07.739648 (+ 885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:07.739653 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:07.739657 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:07.739841 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:07.743133 (+ 3292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:07.743138 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:07.743299 (+ 161us) client_negotiation.cc:769] Sending connection context
0812 00:59:07.744655 (+ 1356us) client_negotiation.cc:240] Negotiation successful
0812 00:59:07.744997 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"mutex_wait_us":2376,"spinlock_wait_cycles":50432,"thread_start_us":671,"threads_started":1}
I20260812 00:59:07.746740 2230 heartbeater.cc:511] Master 127.21.201.126:44537 requested a full tablet report, sending...
I20260812 00:59:07.747185 2227 heartbeater.cc:347] Connected to a master server at 127.21.201.124:39009
I20260812 00:59:07.747371 2227 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:07.747900 2227 heartbeater.cc:511] Master 127.21.201.124:39009 requested a full tablet report, sending...
I20260812 00:59:07.748633 354 ts_manager.cc:194] Re-registered known tserver with Master: 6f446bf9ca014a99b3ddcb2f0e2bc828 (127.21.201.66:42819)
I20260812 00:59:07.749115 1687 ts_manager.cc:194] Registered new tserver with Master: 6f446bf9ca014a99b3ddcb2f0e2bc828 (127.21.201.66:42819)
I20260812 00:59:07.749310 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:07.709105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:39259 (local address 127.21.201.66:55639)
0812 00:59:07.709850 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:07.709882 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:07.710873 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:07.711987 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:07.712001 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:07.713357 (+ 1356us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:07.714753 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:07.714770 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.721755 (+ 6985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.721763 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:07.724273 (+ 2510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:07.724291 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.725690 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.725697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:07.725899 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:07.727405 (+ 1506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:07.727433 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:07.736715 (+ 9282us) spinlock_profiling.cc:243] Waited 4.08 ms on lock 0x60c000034ab8. stack: 00007f47a3f6a66e 00007f47a3f6a430 00007f47a2ff15ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f47a3f600fd 00007f47a3f60977 00007f47aaa7769d 00007f47aa977a15 00007f47aa9734e8 00007f47aa96e33a 00007f47aa9f5dca 00007f47aa9f3f68 00007f47aaa2782f 00007f47aaa2758d
0812 00:59:07.736794 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:07.744252 (+ 7458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:07.744257 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:07.744260 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:07.745654 (+ 1394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:07.747135 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:07.747140 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:07.747143 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:07.747362 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:07.748226 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:07.748231 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:07.748404 (+ 173us) client_negotiation.cc:769] Sending connection context
0812 00:59:07.748565 (+ 161us) client_negotiation.cc:240] Negotiation successful
0812 00:59:07.748888 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":517,"spinlock_wait_cycles":9395456,"thread_start_us":400,"threads_started":1}
I20260812 00:59:07.749892 354 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.201.66@KRBTEST.COM'} at 127.21.201.66:56981
I20260812 00:59:07.749961 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:07.709465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:55639 (local address 127.21.201.125:39259)
0812 00:59:07.710041 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:07.710047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:07.710067 (+ 20us) server_negotiation.cc:407] Connection header received
0812 00:59:07.711283 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:07.711287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:07.711388 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:07.711496 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:07.715028 (+ 3532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.716655 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.724478 (+ 7823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.725232 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.725337 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:07.737120 (+ 11783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:07.737131 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:07.737137 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:07.737193 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:07.742369 (+ 5176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:07.746703 (+ 4334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:07.746714 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:07.746719 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:07.746964 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:07.747520 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:07.747527 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:07.747530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:07.747970 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:07.748122 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:07.748809 (+ 687us) server_negotiation.cc:299] Negotiation successful
0812 00:59:07.749087 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":288,"threads_started":1}
I20260812 00:59:07.750986 2233 heartbeater.cc:347] Connected to a master server at 127.21.201.125:39259
I20260812 00:59:07.751139 2233 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:07.752115 2233 heartbeater.cc:511] Master 127.21.201.125:39259 requested a full tablet report, sending...
I20260812 00:59:07.753196 1351 ts_manager.cc:194] Registered new tserver with Master: 6f446bf9ca014a99b3ddcb2f0e2bc828 (127.21.201.66:42819)
I20260812 00:59:07.754602 2247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:07.711854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58338 (local address 127.21.201.67:36707)
0812 00:59:07.724479 (+ 12625us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:07.724489 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:07.725695 (+ 1206us) server_negotiation.cc:407] Connection header received
0812 00:59:07.725779 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:07.725791 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:07.726098 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:07.726268 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:07.729141 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.731150 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.733332 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:07.734352 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:07.734508 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:07.742479 (+ 7971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:07.742504 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:07.742522 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:07.742574 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:07.751091 (+ 8517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:07.752037 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:07.752046 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:07.752052 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:07.752239 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:07.752717 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:07.752722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:07.752725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:07.753113 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:07.753367 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:07.753898 (+ 531us) server_negotiation.cc:299] Negotiation successful
0812 00:59:07.754201 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12378,"thread_start_us":13473,"threads_started":1}
I20260812 00:59:07.755980 1092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:07.845990 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 876
I20260812 00:59:07.866029 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:36707
--local_ip_for_outbound_sockets=127.21.201.67
--tserver_master_addrs=127.21.201.126:44537
--webserver_port=33361
--webserver_interface=127.21.201.67
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.67.keytab
--principal=oryx/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:41999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.201.126:44537,127.21.201.125:39259
--tserver_master_addrs=127.21.201.126:44537,127.21.201.125:39259,127.21.201.124:39009 with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:59:08.052680 1986 heartbeater.cc:503] Master 127.21.201.126:44537 was elected leader, sending a full tablet report...
W20260812 00:59:08.248729 2250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:08.249087 2250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:08.249194 2250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:08.257243 2250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:08.257385 2250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:08.257560 2250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 00:59:08.268155 2250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:41999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.67.keytab
--principal=oryx/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:36707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=33361
--tserver_master_addrs=127.21.201.126:44537,127.21.201.125:39259,127.21.201.124:39009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:08.269624 2250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:08.270992 2250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:08.286365 2262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:08.286410 2263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:08.286334 2265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:08.287533 2250 server_base.cc:1034] running on GCE node
I20260812 00:59:08.288317 2250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:08.289582 2250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:08.290876 2250 hybrid_clock.cc:648] HybridClock initialized: now 1786496348290822 us; error 46 us; skew 500 ppm
I20260812 00:59:08.291291 2250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:08 dist-test-slave-7f01 krb5kdc[32752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496348, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:08.295480 2250 init.cc:377] Logged in from keytab as oryx/127.21.201.67@KRBTEST.COM (short username oryx)
I20260812 00:59:08.297452 2250 webserver.cc:460] Webserver started at http://127.21.201.67:33361/ using document root <none> and password file <none>
I20260812 00:59:08.298260 2250 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:08.298444 2250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:08.304551 2250 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 00:59:08.307657 2277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:08.309276 2250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 00:59:08.309512 2250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "3a45e74843e94221ad9517a04509d375"
format_stamp: "Formatted at 2026-08-12 00:59:04 on dist-test-slave-7f01"
server_key: "d7eddd017f71f6fa27ee6b2516dadffd"
server_key_iv: "6bcde8b95b42005cdf6818f605931a35"
server_key_version: "encryptionkey@0"
I20260812 00:59:08.310624 2250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:08.335981 2250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:08.342167 2250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:08.342528 2250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:08.344377 2250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:59:08.346652 2250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:59:08.346835 2250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:08.346995 2250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:59:08.347198 2250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:08.389501 2250 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:36707
I20260812 00:59:08.389597 2493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:36707 every 8 connection(s)
I20260812 00:59:08.391112 2250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 00:59:08.401211 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 2250
Aug 12 00:59:08 dist-test-slave-7f01 krb5kdc[32752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496348, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.124@KRBTEST.COM
Aug 12 00:59:08 dist-test-slave-7f01 krb5kdc[32752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496348, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.125@KRBTEST.COM
Aug 12 00:59:08 dist-test-slave-7f01 krb5kdc[32752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496348, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM
I20260812 00:59:08.452636 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:08.394531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:42699 (local address 127.21.201.125:39259)
0812 00:59:08.408282 (+ 13751us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:08.408290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:08.409823 (+ 1533us) server_negotiation.cc:407] Connection header received
0812 00:59:08.412000 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:08.412006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:08.412105 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:08.412229 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:08.414287 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.415471 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.418668 (+ 3197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.419596 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.419776 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:08.433325 (+ 13549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:08.433335 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:08.433340 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:08.433391 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:08.441431 (+ 8040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:08.443361 (+ 1930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:08.443370 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:08.443376 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:08.443620 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:08.448234 (+ 4614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:08.448281 (+ 47us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:08.448286 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:08.448982 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:08.449315 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:08.451898 (+ 2583us) server_negotiation.cc:299] Negotiation successful
0812 00:59:08.452186 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13624,"thread_start_us":13505,"threads_started":1}
I20260812 00:59:08.453245 2501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:08.394653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:52471 (local address 127.21.201.124:39009)
0812 00:59:08.407826 (+ 13173us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:08.407837 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:08.407870 (+ 33us) server_negotiation.cc:407] Connection header received
0812 00:59:08.411241 (+ 3371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:08.411251 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:08.411356 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:08.411468 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:08.414248 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.415471 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.423963 (+ 8492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.424939 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.425077 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:08.431990 (+ 6913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:08.431998 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:08.432003 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:08.432052 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:08.439325 (+ 7273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:08.444380 (+ 5055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:08.444389 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:08.444396 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:08.444576 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:08.447021 (+ 2445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:08.447028 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:08.447032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:08.447502 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:08.447719 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:08.451158 (+ 3439us) server_negotiation.cc:299] Negotiation successful
0812 00:59:08.452845 (+ 1687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1349,"server-negotiator.queue_time_us":12966,"thread_start_us":13510,"threads_started":1}
I20260812 00:59:08.454159 2517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:08.395225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:39259 (local address 127.21.201.67:42699)
0812 00:59:08.409103 (+ 13878us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:08.409130 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:08.410838 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:08.412287 (+ 1449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:08.412295 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:08.413615 (+ 1320us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:08.414079 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:08.414091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.415720 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.415725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:08.418422 (+ 2697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:08.418445 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.420176 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.420186 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:08.420586 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:08.427042 (+ 6456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:08.427093 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:08.433075 (+ 5982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:08.442176 (+ 9101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:08.442189 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:08.442210 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:08.443177 (+ 967us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:08.447832 (+ 4655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:08.447837 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:08.447840 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:08.448030 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:08.451326 (+ 3296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:08.451331 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:08.451517 (+ 186us) client_negotiation.cc:769] Sending connection context
0812 00:59:08.451688 (+ 171us) client_negotiation.cc:240] Negotiation successful
0812 00:59:08.452061 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13600,"mutex_wait_us":118,"thread_start_us":10051,"threads_started":1}
I20260812 00:59:08.456988 2496 heartbeater.cc:347] Connected to a master server at 127.21.201.125:39259
I20260812 00:59:08.457401 2496 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:08.457650 2508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:08.407197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.21.201.67:53339)
0812 00:59:08.409599 (+ 2402us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:08.409623 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:08.410832 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:08.411812 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:08.411826 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:08.413832 (+ 2006us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:08.414377 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:08.414392 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.426751 (+ 12359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.426760 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:08.430223 (+ 3463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:08.430244 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.431945 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.431951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:08.432156 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:08.432946 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:08.432971 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:08.435905 (+ 2934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:08.453142 (+ 17237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:08.453148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:08.453152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:08.453731 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:08.455086 (+ 1355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:08.455094 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:08.455098 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:08.455370 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:08.456212 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:08.456217 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:08.456465 (+ 248us) client_negotiation.cc:769] Sending connection context
0812 00:59:08.456668 (+ 203us) client_negotiation.cc:240] Negotiation successful
0812 00:59:08.457095 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2259,"mutex_wait_us":30,"thread_start_us":1419,"threads_started":1}
I20260812 00:59:08.458392 2510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:08.407332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:53339 (local address 127.21.201.126:44537)
0812 00:59:08.408672 (+ 1340us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:08.408679 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:08.409760 (+ 1081us) server_negotiation.cc:407] Connection header received
0812 00:59:08.411227 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:08.411237 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:08.411356 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:08.411498 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:08.414580 (+ 3082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.419575 (+ 4995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.430458 (+ 10883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.431407 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.431568 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:08.436696 (+ 5128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:08.436706 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:08.436713 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:08.436763 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:08.446012 (+ 9249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:08.454630 (+ 8618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:08.454639 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:08.454643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:08.454840 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:08.455595 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:08.455601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:08.455604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:08.456043 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:08.456216 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:08.457205 (+ 989us) server_negotiation.cc:299] Negotiation successful
0812 00:59:08.457569 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1167,"thread_start_us":1031,"threads_started":1}
I20260812 00:59:08.458496 2496 heartbeater.cc:511] Master 127.21.201.125:39259 requested a full tablet report, sending...
I20260812 00:59:08.459324 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:08.395225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:39009 (local address 127.21.201.67:52471)
0812 00:59:08.406101 (+ 10876us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:08.406188 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:08.410832 (+ 4644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:08.411812 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:08.411826 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:08.412946 (+ 1120us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:08.413997 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:08.414035 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.415786 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.415792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:08.423664 (+ 7872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:08.423690 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.425615 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.425624 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:08.425882 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:08.427830 (+ 1948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:08.427862 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:08.431615 (+ 3753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:08.443389 (+ 11774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:08.443397 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:08.443403 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:08.444172 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:08.446475 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:08.446481 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:08.446484 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:08.446840 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:08.448210 (+ 1370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:08.448224 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:08.448892 (+ 668us) client_negotiation.cc:769] Sending connection context
0812 00:59:08.449291 (+ 399us) client_negotiation.cc:240] Negotiation successful
0812 00:59:08.450107 (+ 816us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10404,"mutex_wait_us":9,"spinlock_wait_cycles":96000,"thread_start_us":13892,"threads_started":1}
I20260812 00:59:08.460750 2495 heartbeater.cc:347] Connected to a master server at 127.21.201.126:44537
I20260812 00:59:08.460898 1351 ts_manager.cc:194] Registered new tserver with Master: 3a45e74843e94221ad9517a04509d375 (127.21.201.67:36707)
I20260812 00:59:08.460965 2495 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:08.461591 2495 heartbeater.cc:511] Master 127.21.201.126:44537 requested a full tablet report, sending...
I20260812 00:59:08.462816 354 ts_manager.cc:194] Re-registered known tserver with Master: 3a45e74843e94221ad9517a04509d375 (127.21.201.67:36707)
I20260812 00:59:08.464174 354 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.201.67@KRBTEST.COM'} at 127.21.201.67:53339
I20260812 00:59:08.466727 2494 heartbeater.cc:347] Connected to a master server at 127.21.201.124:39009
I20260812 00:59:08.466921 2494 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:08.467428 2494 heartbeater.cc:511] Master 127.21.201.124:39009 requested a full tablet report, sending...
I20260812 00:59:08.469101 1687 ts_manager.cc:194] Registered new tserver with Master: 3a45e74843e94221ad9517a04509d375 (127.21.201.67:36707)
I20260812 00:59:08.473490 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:08.416424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55394 (local address 127.21.201.126:44537)
0812 00:59:08.434951 (+ 18527us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:08.434958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:08.434986 (+ 28us) server_negotiation.cc:407] Connection header received
0812 00:59:08.435043 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:08.435048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:08.436054 (+ 1006us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:08.436210 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:08.441694 (+ 5484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.444039 (+ 2345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.447656 (+ 3617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.449762 (+ 2106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.449865 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:08.464173 (+ 14308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:08.464178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:08.464182 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:08.464213 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:08.468089 (+ 3876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:08.469853 (+ 1764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:08.469859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:08.469863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:08.470050 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:08.471001 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:08.471010 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:08.471016 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:08.471485 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:08.471821 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:08.472735 (+ 914us) server_negotiation.cc:299] Negotiation successful
0812 00:59:08.473027 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18370,"thread_start_us":18238,"threads_started":1}
I20260812 00:59:08.477526 2533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:08.416964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36374 (local address 127.21.201.124:39009)
0812 00:59:08.435992 (+ 19028us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:08.435998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:08.442204 (+ 6206us) server_negotiation.cc:407] Connection header received
0812 00:59:08.444512 (+ 2308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:08.444518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:08.444593 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:08.444691 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:08.451068 (+ 6377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.452497 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.455565 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.456254 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.456350 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:08.464715 (+ 8365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:08.464723 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:08.464729 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:08.464797 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:08.472772 (+ 7975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:08.474261 (+ 1489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:08.474266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:08.474268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:08.474481 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:08.475033 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:08.475038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:08.475041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:08.475492 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:08.475723 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:08.476574 (+ 851us) server_negotiation.cc:299] Negotiation successful
0812 00:59:08.476851 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":18864,"thread_start_us":18712,"threads_started":1}
I20260812 00:59:08.481284 2539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:08.440694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55770 (local address 127.21.201.125:39259)
0812 00:59:08.451146 (+ 10452us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:08.451151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:08.461238 (+ 10087us) server_negotiation.cc:407] Connection header received
0812 00:59:08.461651 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:08.461658 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:08.461732 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:08.461817 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:08.465417 (+ 3600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.466680 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.468636 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.469555 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.469712 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:08.473096 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:08.473104 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:08.473109 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:08.473155 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:08.477238 (+ 4083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:08.478395 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:08.478403 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:08.478409 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:08.478593 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:08.479085 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:08.479090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:08.479093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:08.479486 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:08.479880 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:08.480455 (+ 575us) server_negotiation.cc:299] Negotiation successful
0812 00:59:08.480861 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10306,"thread_start_us":10143,"threads_started":1}
I20260812 00:59:08.479063 354 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:59:08.487636 354 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:59:08.523885 2564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:08.510223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:36707 (local address 127.0.0.1:58342)
0812 00:59:08.514350 (+ 4127us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:08.514368 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:08.514591 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:08.515864 (+ 1273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:08.515872 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:08.515901 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:08.516365 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:08.516379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.518892 (+ 2513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.518897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:08.521063 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:08.521079 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.522895 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.522903 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:08.523089 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:08.523124 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 00:59:08.523217 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 00:59:08.523362 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4001,"mutex_wait_us":56,"thread_start_us":3779,"threads_started":1}
I20260812 00:59:08.523948 2569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:08.511232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58342 (local address 127.21.201.67:36707)
0812 00:59:08.512116 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:08.512129 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:08.514473 (+ 2344us) server_negotiation.cc:407] Connection header received
0812 00:59:08.514816 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:08.514825 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:08.515164 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:08.515366 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:08.516596 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.518504 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.521253 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.522513 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.522664 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:08.522912 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:08.523344 (+ 432us) server_negotiation.cc:299] Negotiation successful
0812 00:59:08.523551 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":658,"threads_started":1}
I20260812 00:59:08.525645 2561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:08.509159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:42819 (local address 127.0.0.1:35752)
0812 00:59:08.510415 (+ 1256us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:08.510442 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:08.510612 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:08.517634 (+ 7022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:08.517639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:08.517664 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:08.518097 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:08.518111 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.520263 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.520269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:08.522544 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:08.522557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.524793 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.524800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:08.524982 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:08.525017 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 00:59:08.525115 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 00:59:08.525250 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"mutex_wait_us":92,"thread_start_us":2992,"threads_started":1}
I20260812 00:59:08.526154 2562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:08.509433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35752 (local address 127.21.201.66:42819)
0812 00:59:08.517054 (+ 7621us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:08.517066 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:08.517092 (+ 26us) server_negotiation.cc:407] Connection header received
0812 00:59:08.517189 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:08.517195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:08.517405 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:08.517639 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:08.518535 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.520103 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.523009 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.524582 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.524693 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:08.524844 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:08.525342 (+ 498us) server_negotiation.cc:299] Negotiation successful
0812 00:59:08.525653 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7449,"thread_start_us":7360,"threads_started":1}
I20260812 00:59:08.527835 2567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:08.510287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:37425 (local address 127.0.0.1:58416)
0812 00:59:08.516430 (+ 6143us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:08.516449 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:08.516560 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:08.517186 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:08.517191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:08.517219 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:08.517562 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:08.517574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.519376 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.519384 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:08.522818 (+ 3434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:08.522831 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.526959 (+ 4128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.526968 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:08.527170 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:08.527202 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 00:59:08.527310 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 00:59:08.527442 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5994,"mutex_wait_us":1,"thread_start_us":5810,"threads_started":1}
I20260812 00:59:08.528033 2572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:08.511703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58416 (local address 127.21.201.65:37425)
0812 00:59:08.515917 (+ 4214us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:08.515926 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:08.516573 (+ 647us) server_negotiation.cc:407] Connection header received
0812 00:59:08.516688 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:08.516700 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:08.516968 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:08.517168 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:08.517808 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.519233 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.523349 (+ 4116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.526015 (+ 2666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.526226 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:08.526651 (+ 425us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:08.527429 (+ 778us) server_negotiation.cc:299] Negotiation successful
0812 00:59:08.527652 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3996,"thread_start_us":3982,"threads_started":1}
I20260812 00:59:08.528815 2370 tablet_service.cc:1467] Processing CreateTablet for tablet a48a4fec5edc4997b5fd68aed1645a06 (DEFAULT_TABLE table=test-table [id=b8dfd4b4c52a42c8bd912e984beb2054]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:59:08.530671 2370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48a4fec5edc4997b5fd68aed1645a06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:08.532541 2108 tablet_service.cc:1467] Processing CreateTablet for tablet a48a4fec5edc4997b5fd68aed1645a06 (DEFAULT_TABLE table=test-table [id=b8dfd4b4c52a42c8bd912e984beb2054]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:59:08.532833 1883 tablet_service.cc:1467] Processing CreateTablet for tablet a48a4fec5edc4997b5fd68aed1645a06 (DEFAULT_TABLE table=test-table [id=b8dfd4b4c52a42c8bd912e984beb2054]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:59:08.534051 2108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48a4fec5edc4997b5fd68aed1645a06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:08.534086 1883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48a4fec5edc4997b5fd68aed1645a06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:08.547840 2573 tablet_bootstrap.cc:492] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e: Bootstrap starting.
I20260812 00:59:08.551146 2576 tablet_bootstrap.cc:492] T a48a4fec5edc4997b5fd68aed1645a06 P 6f446bf9ca014a99b3ddcb2f0e2bc828: Bootstrap starting.
I20260812 00:59:08.554160 2573 tablet_bootstrap.cc:654] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:08.554706 2575 tablet_bootstrap.cc:492] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375: Bootstrap starting.
I20260812 00:59:08.554836 2576 tablet_bootstrap.cc:654] T a48a4fec5edc4997b5fd68aed1645a06 P 6f446bf9ca014a99b3ddcb2f0e2bc828: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:08.555863 2573 log.cc:824] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e: Log is configured to *not* fsync() on all Append() calls
I20260812 00:59:08.556209 2576 log.cc:824] T a48a4fec5edc4997b5fd68aed1645a06 P 6f446bf9ca014a99b3ddcb2f0e2bc828: Log is configured to *not* fsync() on all Append() calls
I20260812 00:59:08.558130 2575 tablet_bootstrap.cc:654] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:08.559263 2575 log.cc:824] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375: Log is configured to *not* fsync() on all Append() calls
I20260812 00:59:08.559577 2573 tablet_bootstrap.cc:492] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e: No bootstrap required, opened a new log
I20260812 00:59:08.559954 2573 ts_tablet_manager.cc:1397] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20260812 00:59:08.564774 2575 tablet_bootstrap.cc:492] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375: No bootstrap required, opened a new log
I20260812 00:59:08.565085 2575 ts_tablet_manager.cc:1397] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260812 00:59:08.566449 2576 tablet_bootstrap.cc:492] T a48a4fec5edc4997b5fd68aed1645a06 P 6f446bf9ca014a99b3ddcb2f0e2bc828: No bootstrap required, opened a new log
I20260812 00:59:08.566805 2576 ts_tablet_manager.cc:1397] T a48a4fec5edc4997b5fd68aed1645a06 P 6f446bf9ca014a99b3ddcb2f0e2bc828: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.004s
I20260812 00:59:08.570111 2573 raft_consensus.cc:348] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42819 } } peers { permanent_uuid: "3a45e74843e94221ad9517a04509d375" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 36707 } } peers { permanent_uuid: "2ac902d212d8411bb6adfa954008ed8e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 37425 } }
I20260812 00:59:08.570643 2573 raft_consensus.cc:374] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:08.570753 2573 raft_consensus.cc:729] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac902d212d8411bb6adfa954008ed8e, State: Initialized, Role: FOLLOWER
I20260812 00:59:08.571476 2573 consensus_queue.cc:260] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42819 } } peers { permanent_uuid: "3a45e74843e94221ad9517a04509d375" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 36707 } } peers { permanent_uuid: "2ac902d212d8411bb6adfa954008ed8e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 37425 } }
I20260812 00:59:08.574811 2575 raft_consensus.cc:348] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42819 } } peers { permanent_uuid: "3a45e74843e94221ad9517a04509d375" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 36707 } } peers { permanent_uuid: "2ac902d212d8411bb6adfa954008ed8e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 37425 } }
I20260812 00:59:08.575294 2575 raft_consensus.cc:374] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:08.575394 2575 raft_consensus.cc:729] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a45e74843e94221ad9517a04509d375, State: Initialized, Role: FOLLOWER
I20260812 00:59:08.576318 2575 consensus_queue.cc:260] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42819 } } peers { permanent_uuid: "3a45e74843e94221ad9517a04509d375" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 36707 } } peers { permanent_uuid: "2ac902d212d8411bb6adfa954008ed8e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 37425 } }
I20260812 00:59:08.578275 2576 raft_consensus.cc:348] T a48a4fec5edc4997b5fd68aed1645a06 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42819 } } peers { permanent_uuid: "3a45e74843e94221ad9517a04509d375" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 36707 } } peers { permanent_uuid: "2ac902d212d8411bb6adfa954008ed8e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 37425 } }
I20260812 00:59:08.578799 2576 raft_consensus.cc:374] T a48a4fec5edc4997b5fd68aed1645a06 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:08.578923 2576 raft_consensus.cc:729] T a48a4fec5edc4997b5fd68aed1645a06 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f446bf9ca014a99b3ddcb2f0e2bc828, State: Initialized, Role: FOLLOWER
I20260812 00:59:08.579113 2575 ts_tablet_manager.cc:1428] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375: Time spent starting tablet: real 0.014s user 0.006s sys 0.007s
I20260812 00:59:08.579396 2495 heartbeater.cc:503] Master 127.21.201.126:44537 was elected leader, sending a full tablet report...
I20260812 00:59:08.579702 2576 consensus_queue.cc:260] T a48a4fec5edc4997b5fd68aed1645a06 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42819 } } peers { permanent_uuid: "3a45e74843e94221ad9517a04509d375" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 36707 } } peers { permanent_uuid: "2ac902d212d8411bb6adfa954008ed8e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 37425 } }
I20260812 00:59:08.580405 2573 ts_tablet_manager.cc:1428] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e: Time spent starting tablet: real 0.020s user 0.007s sys 0.006s
I20260812 00:59:08.586899 2576 ts_tablet_manager.cc:1428] T a48a4fec5edc4997b5fd68aed1645a06 P 6f446bf9ca014a99b3ddcb2f0e2bc828: Time spent starting tablet: real 0.020s user 0.015s sys 0.003s
I20260812 00:59:08.587220 2230 heartbeater.cc:503] Master 127.21.201.126:44537 was elected leader, sending a full tablet report...
W20260812 00:59:08.644552 2497 tablet.cc:2369] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:59:08.708119 2234 tablet.cc:2369] T a48a4fec5edc4997b5fd68aed1645a06 P 6f446bf9ca014a99b3ddcb2f0e2bc828: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:59:08.719187 2580 raft_consensus.cc:482] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:59:08.719601 2580 raft_consensus.cc:504] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42819 } } peers { permanent_uuid: "3a45e74843e94221ad9517a04509d375" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 36707 } } peers { permanent_uuid: "2ac902d212d8411bb6adfa954008ed8e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 37425 } }
I20260812 00:59:08.721668 2580 leader_election.cc:288] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f446bf9ca014a99b3ddcb2f0e2bc828 (127.21.201.66:42819), 3a45e74843e94221ad9517a04509d375 (127.21.201.67:36707)
I20260812 00:59:08.730103 2562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:08.722024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:59649 (local address 127.21.201.66:42819)
0812 00:59:08.722300 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:08.722309 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:08.723442 (+ 1133us) server_negotiation.cc:407] Connection header received
0812 00:59:08.723605 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:08.723610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:08.723703 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:08.723827 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:08.724493 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.725305 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.727719 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.728890 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.729083 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:08.729163 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:08.729678 (+ 515us) server_negotiation.cc:299] Negotiation successful
0812 00:59:08.729824 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 00:59:08.730314 2569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:08.722224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:57337 (local address 127.21.201.67:36707)
0812 00:59:08.722505 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:08.722513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:08.723344 (+ 831us) server_negotiation.cc:407] Connection header received
0812 00:59:08.723639 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:08.723643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:08.723705 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:08.723853 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:08.724542 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.725454 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.727844 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.729198 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.729379 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:08.729451 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:08.729914 (+ 463us) server_negotiation.cc:299] Negotiation successful
0812 00:59:08.730070 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 00:59:08.730901 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:08.722065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:42819 (local address 127.21.201.65:59649)
0812 00:59:08.723274 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:08.723321 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:08.723462 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:08.723849 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:08.723853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:08.723878 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:08.724321 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:08.724333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.725454 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.725460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:08.727545 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:08.727562 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.729109 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.729116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:08.729459 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:08.729530 (+ 71us) client_negotiation.cc:769] Sending connection context
0812 00:59:08.729650 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 00:59:08.729877 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1093,"mutex_wait_us":6,"thread_start_us":725,"threads_started":1}
I20260812 00:59:08.731211 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:08.722014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:36707 (local address 127.21.201.65:57337)
0812 00:59:08.723211 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:08.723229 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:08.723442 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:08.723896 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:08.723901 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:08.723924 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:08.724371 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:08.724380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.725580 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.725583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:08.727596 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:08.727610 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.729480 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.729486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:08.729736 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:08.729766 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 00:59:08.730580 (+ 814us) client_negotiation.cc:240] Negotiation successful
0812 00:59:08.730762 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077,"mutex_wait_us":26,"thread_start_us":628,"threads_started":1}
I20260812 00:59:08.732646 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48a4fec5edc4997b5fd68aed1645a06" candidate_uuid: "2ac902d212d8411bb6adfa954008ed8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" is_pre_election: true
I20260812 00:59:08.732663 2402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48a4fec5edc4997b5fd68aed1645a06" candidate_uuid: "2ac902d212d8411bb6adfa954008ed8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a45e74843e94221ad9517a04509d375" is_pre_election: true
I20260812 00:59:08.733254 2145 raft_consensus.cc:2454] T a48a4fec5edc4997b5fd68aed1645a06 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ac902d212d8411bb6adfa954008ed8e in term 0.
I20260812 00:59:08.733397 2402 raft_consensus.cc:2454] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ac902d212d8411bb6adfa954008ed8e in term 0.
I20260812 00:59:08.733868 2588 raft_consensus.cc:482] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:59:08.734184 1801 leader_election.cc:302] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ac902d212d8411bb6adfa954008ed8e, 6f446bf9ca014a99b3ddcb2f0e2bc828; no voters:
I20260812 00:59:08.734109 2588 raft_consensus.cc:504] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42819 } } peers { permanent_uuid: "3a45e74843e94221ad9517a04509d375" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 36707 } } peers { permanent_uuid: "2ac902d212d8411bb6adfa954008ed8e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 37425 } }
I20260812 00:59:08.735987 2580 raft_consensus.cc:2781] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:59:08.736074 2588 leader_election.cc:288] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f446bf9ca014a99b3ddcb2f0e2bc828 (127.21.201.66:42819), 2ac902d212d8411bb6adfa954008ed8e (127.21.201.65:37425)
I20260812 00:59:08.736179 2580 raft_consensus.cc:482] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:59:08.736315 2580 raft_consensus.cc:3037] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:08.739161 2580 raft_consensus.cc:504] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42819 } } peers { permanent_uuid: "3a45e74843e94221ad9517a04509d375" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 36707 } } peers { permanent_uuid: "2ac902d212d8411bb6adfa954008ed8e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 37425 } }
I20260812 00:59:08.740020 2580 leader_election.cc:288] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e [CANDIDATE]: Term 1 election: Requested vote from peers 6f446bf9ca014a99b3ddcb2f0e2bc828 (127.21.201.66:42819), 3a45e74843e94221ad9517a04509d375 (127.21.201.67:36707)
I20260812 00:59:08.741007 2402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48a4fec5edc4997b5fd68aed1645a06" candidate_uuid: "2ac902d212d8411bb6adfa954008ed8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a45e74843e94221ad9517a04509d375"
I20260812 00:59:08.740990 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48a4fec5edc4997b5fd68aed1645a06" candidate_uuid: "2ac902d212d8411bb6adfa954008ed8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828"
I20260812 00:59:08.741261 2402 raft_consensus.cc:3037] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:08.741251 2145 raft_consensus.cc:3037] T a48a4fec5edc4997b5fd68aed1645a06 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:08.745291 2145 raft_consensus.cc:2454] T a48a4fec5edc4997b5fd68aed1645a06 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ac902d212d8411bb6adfa954008ed8e in term 1.
I20260812 00:59:08.745307 2402 raft_consensus.cc:2454] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ac902d212d8411bb6adfa954008ed8e in term 1.
I20260812 00:59:08.745863 1801 leader_election.cc:302] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ac902d212d8411bb6adfa954008ed8e, 6f446bf9ca014a99b3ddcb2f0e2bc828; no voters:
I20260812 00:59:08.746097 2508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:08.738328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:37425 (local address 127.21.201.67:43949)
0812 00:59:08.738553 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:08.738568 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:08.738657 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:08.739155 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:08.739160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:08.739187 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:08.739439 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:08.739456 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.740983 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.740993 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:08.743236 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:08.743253 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.745159 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.745165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:08.745487 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:08.745526 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 00:59:08.745635 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 00:59:08.745815 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260812 00:59:08.746484 2580 raft_consensus.cc:2781] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:59:08.746836 2572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:08.736773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:43949 (local address 127.21.201.65:37425)
0812 00:59:08.736967 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:08.736973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:08.738751 (+ 1778us) server_negotiation.cc:407] Connection header received
0812 00:59:08.738827 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:08.738832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:08.738906 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:08.738980 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:08.739633 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.740810 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.743439 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.744766 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.744936 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:08.745015 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:08.746399 (+ 1384us) server_negotiation.cc:299] Negotiation successful
0812 00:59:08.746556 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 00:59:08.747642 1915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48a4fec5edc4997b5fd68aed1645a06" candidate_uuid: "3a45e74843e94221ad9517a04509d375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac902d212d8411bb6adfa954008ed8e" is_pre_election: true
I20260812 00:59:08.748744 2580 raft_consensus.cc:686] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e [term 1 LEADER]: Becoming Leader. State: Replica: 2ac902d212d8411bb6adfa954008ed8e, State: Running, Role: LEADER
I20260812 00:59:08.750099 2580 consensus_queue.cc:237] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42819 } } peers { permanent_uuid: "3a45e74843e94221ad9517a04509d375" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 36707 } } peers { permanent_uuid: "2ac902d212d8411bb6adfa954008ed8e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 37425 } }
I20260812 00:59:08.753726 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:08.737288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:42819 (local address 127.21.201.67:33301)
0812 00:59:08.737888 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:08.737915 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:08.738042 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:08.746078 (+ 8036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:08.746089 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:08.746121 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:08.746376 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:08.746390 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.748435 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.748443 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:08.750980 (+ 2537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:08.751000 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.752751 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.752759 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:08.753057 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:08.753125 (+ 68us) client_negotiation.cc:769] Sending connection context
0812 00:59:08.753239 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 00:59:08.753402 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260812 00:59:08.753954 2562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:08.736908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:33301 (local address 127.21.201.66:42819)
0812 00:59:08.745720 (+ 8812us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:08.745726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:08.745746 (+ 20us) server_negotiation.cc:407] Connection header received
0812 00:59:08.745807 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:08.745811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:08.745896 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:08.746549 (+ 653us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:08.746698 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.748240 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.751236 (+ 2996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.752515 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.752738 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:08.752851 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:08.753303 (+ 452us) server_negotiation.cc:299] Negotiation successful
0812 00:59:08.753498 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8679}
I20260812 00:59:08.754768 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48a4fec5edc4997b5fd68aed1645a06" candidate_uuid: "3a45e74843e94221ad9517a04509d375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" is_pre_election: true
I20260812 00:59:08.755182 2145 raft_consensus.cc:2379] T a48a4fec5edc4997b5fd68aed1645a06 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a45e74843e94221ad9517a04509d375 in current term 1: Already voted for candidate 2ac902d212d8411bb6adfa954008ed8e in this term.
I20260812 00:59:08.756800 2290 leader_election.cc:302] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a45e74843e94221ad9517a04509d375; no voters: 2ac902d212d8411bb6adfa954008ed8e, 6f446bf9ca014a99b3ddcb2f0e2bc828
W20260812 00:59:08.757318 1988 tablet.cc:2369] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:59:08.757573 2588 raft_consensus.cc:2726] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 00:59:08.759137 354 catalog_manager.cc:5495] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ac902d212d8411bb6adfa954008ed8e (127.21.201.65). New cstate: current_term: 1 leader_uuid: "2ac902d212d8411bb6adfa954008ed8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a45e74843e94221ad9517a04509d375" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 36707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ac902d212d8411bb6adfa954008ed8e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 37425 } health_report { overall_health: HEALTHY } } }
I20260812 00:59:08.833395 2572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:08.825990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58432 (local address 127.21.201.65:37425)
0812 00:59:08.826187 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:08.826192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:08.826211 (+ 19us) server_negotiation.cc:407] Connection header received
0812 00:59:08.826323 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:08.826327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:08.826433 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:08.826573 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:59:08.827472 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.828603 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.830454 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.831121 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.831234 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:08.832137 (+ 903us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:59:08.832702 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:59:08.832823 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:08.833019 (+ 196us) server_negotiation.cc:299] Negotiation successful
0812 00:59:08.833106 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 00:59:08.843611 2145 raft_consensus.cc:1261] T a48a4fec5edc4997b5fd68aed1645a06 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [term 1 FOLLOWER]: Refusing update from remote peer 2ac902d212d8411bb6adfa954008ed8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:59:08.843657 2402 raft_consensus.cc:1261] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375 [term 1 FOLLOWER]: Refusing update from remote peer 2ac902d212d8411bb6adfa954008ed8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:59:08.844678 2595 consensus_queue.cc:1035] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:59:08.845253 2580 consensus_queue.cc:1035] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a45e74843e94221ad9517a04509d375" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 36707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:59:08.858950 2603 mvcc.cc:205] Tried to move back new op lower bound from 7317489044852498432 to 7317489044485586944. Current Snapshot: MvccSnapshot[applied={T|T < 7317489044852498432}]
I20260812 00:59:08.860803 2609 mvcc.cc:205] Tried to move back new op lower bound from 7317489044852498432 to 7317489044485586944. Current Snapshot: MvccSnapshot[applied={T|T < 7317489044852498432}]
I20260812 00:59:08.883126 354 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:59:08.883438 354 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:59:08.891057 354 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ef13f6098fe4535af6f2557d272fd2b: Sending DeleteTablet for 3 replicas of tablet a48a4fec5edc4997b5fd68aed1645a06
I20260812 00:59:08.892513 2108 tablet_service.cc:1514] Processing DeleteTablet for tablet a48a4fec5edc4997b5fd68aed1645a06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:59:08 UTC) from {username='oryx', principal='oryx/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:35752
I20260812 00:59:08.893328 2370 tablet_service.cc:1514] Processing DeleteTablet for tablet a48a4fec5edc4997b5fd68aed1645a06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:59:08 UTC) from {username='oryx', principal='oryx/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:58342
I20260812 00:59:08.893946 1883 tablet_service.cc:1514] Processing DeleteTablet for tablet a48a4fec5edc4997b5fd68aed1645a06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:59:08 UTC) from {username='oryx', principal='oryx/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:58416
I20260812 00:59:08.900578 2624 tablet_replica.cc:330] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375: stopping tablet replica
I20260812 00:59:08.901826 2623 tablet_replica.cc:330] T a48a4fec5edc4997b5fd68aed1645a06 P 6f446bf9ca014a99b3ddcb2f0e2bc828: stopping tablet replica
I20260812 00:59:08.902451 2623 raft_consensus.cc:2229] T a48a4fec5edc4997b5fd68aed1645a06 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:59:08.903239 2623 raft_consensus.cc:2258] T a48a4fec5edc4997b5fd68aed1645a06 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:59:08.904171 2624 raft_consensus.cc:2229] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:59:08.904429 2625 tablet_replica.cc:330] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e: stopping tablet replica
I20260812 00:59:08.904717 2624 raft_consensus.cc:2258] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:59:08.904906 2625 raft_consensus.cc:2229] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:59:08.905864 2625 raft_consensus.cc:2258] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:59:08.906205 2623 ts_tablet_manager.cc:1905] T a48a4fec5edc4997b5fd68aed1645a06 P 6f446bf9ca014a99b3ddcb2f0e2bc828: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:59:08.906587 2624 ts_tablet_manager.cc:1905] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:59:08.908553 2625 ts_tablet_manager.cc:1905] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:59:08.916569 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 303
I20260812 00:59:08.916671 2623 ts_tablet_manager.cc:1918] T a48a4fec5edc4997b5fd68aed1645a06 P 6f446bf9ca014a99b3ddcb2f0e2bc828: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:59:08.916875 2623 log.cc:1196] T a48a4fec5edc4997b5fd68aed1645a06 P 6f446bf9ca014a99b3ddcb2f0e2bc828: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/a48a4fec5edc4997b5fd68aed1645a06
I20260812 00:59:08.917550 2623 ts_tablet_manager.cc:1939] T a48a4fec5edc4997b5fd68aed1645a06 P 6f446bf9ca014a99b3ddcb2f0e2bc828: Deleting consensus metadata
I20260812 00:59:08.917986 401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:08.920959 2625 ts_tablet_manager.cc:1918] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:59:08.921350 2625 log.cc:1196] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/wal/wals/a48a4fec5edc4997b5fd68aed1645a06
I20260812 00:59:08.922214 2625 ts_tablet_manager.cc:1939] T a48a4fec5edc4997b5fd68aed1645a06 P 2ac902d212d8411bb6adfa954008ed8e: Deleting consensus metadata
I20260812 00:59:08.923844 2624 ts_tablet_manager.cc:1918] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:59:08.924070 2624 log.cc:1196] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/a48a4fec5edc4997b5fd68aed1645a06
I20260812 00:59:08.924773 2624 ts_tablet_manager.cc:1939] T a48a4fec5edc4997b5fd68aed1645a06 P 3a45e74843e94221ad9517a04509d375: Deleting consensus metadata
I20260812 00:59:08.939005 2539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:08.896902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55782 (local address 127.21.201.125:39259)
0812 00:59:08.897189 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:08.897199 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:08.914668 (+ 17469us) server_negotiation.cc:407] Connection header received
0812 00:59:08.914899 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:08.914905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:08.915016 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:08.915133 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:08.925933 (+ 10800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.926981 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.928796 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.929332 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.929479 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:08.933048 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:08.933056 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:08.933061 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:08.933136 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:08.936014 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:08.936787 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:08.936792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:08.936794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:08.936945 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:08.937348 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:08.937360 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:08.937362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:08.937700 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:08.937988 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:08.938360 (+ 372us) server_negotiation.cc:299] Negotiation successful
0812 00:59:08.938604 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260812 00:59:08.953045 2533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:08.915562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36390 (local address 127.21.201.124:39009)
0812 00:59:08.915951 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:08.915960 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:08.939050 (+ 23090us) server_negotiation.cc:407] Connection header received
0812 00:59:08.939639 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:08.939644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:08.939719 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:08.939921 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:08.941420 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.942157 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.943656 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:08.944171 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:08.944322 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:08.947063 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:08.947068 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:08.947072 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:08.947105 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:08.950215 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:08.950948 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:08.950954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:08.950963 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:08.951103 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:08.951488 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:08.951493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:08.951498 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:08.951843 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:08.952119 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:08.952416 (+ 297us) server_negotiation.cc:299] Negotiation successful
0812 00:59:08.952661 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224}
I20260812 00:59:09.088239 327 catalog_manager.cc:4841] TS 3a45e74843e94221ad9517a04509d375 (127.21.201.67:36707): tablet a48a4fec5edc4997b5fd68aed1645a06 (table test-table [id=b8dfd4b4c52a42c8bd912e984beb2054]) successfully deleted
I20260812 00:59:09.089094 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 303
I20260812 00:59:09.156925 2539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:09.137657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55784 (local address 127.21.201.125:39259)
0812 00:59:09.137920 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:09.137929 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:09.138938 (+ 1009us) server_negotiation.cc:407] Connection header received
0812 00:59:09.139484 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:09.139492 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:09.139582 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:09.139687 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:09.141985 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:09.143015 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:09.144855 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:09.145568 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:09.145724 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:09.149408 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:09.149413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:09.149417 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:09.149456 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:09.153512 (+ 4056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:09.154458 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:09.154470 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:09.154473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:09.154632 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:09.155114 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:09.155121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:09.155126 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:09.155474 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:09.155800 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:09.156288 (+ 488us) server_negotiation.cc:299] Negotiation successful
0812 00:59:09.156553 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 00:59:09.159859 2533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:09.136602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36392 (local address 127.21.201.124:39009)
0812 00:59:09.136891 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:09.136901 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:09.138618 (+ 1717us) server_negotiation.cc:407] Connection header received
0812 00:59:09.139068 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:09.139078 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:09.139195 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:09.139281 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:09.141471 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:09.142720 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:09.144984 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:09.145664 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:09.145778 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:09.149891 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:09.149900 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:09.149903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:09.149948 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:09.153742 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:09.156022 (+ 2280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:09.156027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:09.156031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:09.156203 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:09.157645 (+ 1442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:09.157653 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:09.157656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:09.158154 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:09.158447 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:09.159101 (+ 654us) server_negotiation.cc:299] Negotiation successful
0812 00:59:09.159412 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 00:59:09.874698 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:09.872199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.21.201.65:57931)
0812 00:59:09.874241 (+ 2042us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:09.874404 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1892,"thread_start_us":1774,"threads_started":1}
W20260812 00:59:09.875239 1986 heartbeater.cc:644] Failed to heartbeat to 127.21.201.126:44537 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
I20260812 00:59:09.875497 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:09.873827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.21.201.67:52705)
0812 00:59:09.875084 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:09.875242 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":818,"thread_start_us":692,"threads_started":1}
W20260812 00:59:09.875950 2495 heartbeater.cc:644] Failed to heartbeat to 127.21.201.126:44537 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
I20260812 00:59:09.878103 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:09.876596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.21.201.66:60769)
0812 00:59:09.877712 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:09.877869 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1004,"thread_start_us":877,"threads_started":1}
W20260812 00:59:09.878898 2230 heartbeater.cc:644] Failed to heartbeat to 127.21.201.126:44537 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
I20260812 00:59:10.392794 2653 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ef13f6098fe4535af6f2557d272fd2b)
I20260812 00:59:10.393134 2652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ef13f6098fe4535af6f2557d272fd2b)
I20260812 00:59:10.393165 2653 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: false } }
I20260812 00:59:10.393421 2652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: false } }
I20260812 00:59:10.394464 2652 leader_election.cc:288] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ef13f6098fe4535af6f2557d272fd2b (127.21.201.126:44537), c4c07c47632b4011bfc2a2bb1ed4a100 (127.21.201.125:39259)
I20260812 00:59:10.395968 2653 leader_election.cc:288] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ef13f6098fe4535af6f2557d272fd2b (127.21.201.126:44537), 2e5afbe350a9486e800c519579288778 (127.21.201.124:39009)
I20260812 00:59:10.396879 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:10.394873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.0.0.1:55766)
0812 00:59:10.395989 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:10.396307 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":880,"thread_start_us":608,"threads_started":1}
W20260812 00:59:10.397258 1318 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
I20260812 00:59:10.399163 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:10.398719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.0.0.1:55776)
0812 00:59:10.398883 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:10.398972 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260812 00:59:10.399451 1318 leader_election.cc:334] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3ef13f6098fe4535af6f2557d272fd2b (127.21.201.126:44537): Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
I20260812 00:59:10.399600 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:10.394693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.0.0.1:55764)
0812 00:59:10.398937 (+ 4244us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:10.399158 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4119,"thread_start_us":3913,"threads_started":1}
W20260812 00:59:10.400764 1666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
I20260812 00:59:10.406335 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:10.403181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.0.0.1:55784)
0812 00:59:10.405644 (+ 2463us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:10.405881 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2284,"thread_start_us":2127,"threads_started":1}
W20260812 00:59:10.406634 1666 leader_election.cc:334] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3ef13f6098fe4535af6f2557d272fd2b (127.21.201.126:44537): Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
I20260812 00:59:10.408099 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:10.396222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:39009 (local address 127.0.0.1:36402)
0812 00:59:10.396863 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:10.396889 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:10.397065 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:10.401131 (+ 4066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:10.401136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:10.401159 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:10.401493 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:10.401513 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:10.403317 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:10.403324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:10.404783 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:10.404798 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:10.406826 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:10.406833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:10.407215 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:10.407251 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 00:59:10.407409 (+ 158us) client_negotiation.cc:240] Negotiation successful
0812 00:59:10.407590 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":70,"thread_start_us":528,"threads_started":1}
I20260812 00:59:10.408089 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:10.400194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36402 (local address 127.21.201.124:39009)
0812 00:59:10.400799 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:10.400805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:10.400822 (+ 17us) server_negotiation.cc:407] Connection header received
0812 00:59:10.400884 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:10.400889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:10.400988 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:10.401110 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:10.401770 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:10.403157 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:10.404975 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:10.406605 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:10.406824 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:10.406915 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:10.407442 (+ 527us) server_negotiation.cc:299] Negotiation successful
0812 00:59:10.407593 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":474,"thread_start_us":299,"threads_started":1}
I20260812 00:59:10.408574 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:10.399856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:39259 (local address 127.0.0.1:55800)
0812 00:59:10.400114 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:10.400143 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:10.400374 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:10.400815 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:10.400820 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:10.400852 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:10.401925 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:10.401945 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:10.403668 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:10.403677 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:10.406072 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:10.406089 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:10.407703 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:10.407710 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:10.407962 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:10.407990 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 00:59:10.408112 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 00:59:10.408262 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":70}
I20260812 00:59:10.408679 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:10.399211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55800 (local address 127.21.201.125:39259)
0812 00:59:10.400340 (+ 1129us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:10.400349 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:10.400388 (+ 39us) server_negotiation.cc:407] Connection header received
0812 00:59:10.400555 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:10.400561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:10.400642 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:10.400772 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:10.402153 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:10.403406 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:10.406248 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:10.407491 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:10.407717 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:10.407826 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:10.408152 (+ 326us) server_negotiation.cc:299] Negotiation successful
0812 00:59:10.408293 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":988,"thread_start_us":759,"threads_started":1}
I20260812 00:59:10.408939 1703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2e5afbe350a9486e800c519579288778" is_pre_election: true
I20260812 00:59:10.409318 1364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e5afbe350a9486e800c519579288778" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" is_pre_election: true
I20260812 00:59:10.409385 1703 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4c07c47632b4011bfc2a2bb1ed4a100 in term 1.
I20260812 00:59:10.409638 1364 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e5afbe350a9486e800c519579288778 in term 1.
I20260812 00:59:10.410136 1316 leader_election.cc:302] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e5afbe350a9486e800c519579288778, c4c07c47632b4011bfc2a2bb1ed4a100; no voters: 3ef13f6098fe4535af6f2557d272fd2b
I20260812 00:59:10.410609 2653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 00:59:10.410751 2653 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3ef13f6098fe4535af6f2557d272fd2b)
I20260812 00:59:10.410856 2653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 1 FOLLOWER]: Advancing to term 2
I20260812 00:59:10.410974 1666 leader_election.cc:302] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e5afbe350a9486e800c519579288778, c4c07c47632b4011bfc2a2bb1ed4a100; no voters: 3ef13f6098fe4535af6f2557d272fd2b
I20260812 00:59:10.411294 2652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 00:59:10.411438 2652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3ef13f6098fe4535af6f2557d272fd2b)
I20260812 00:59:10.411552 2652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 1 FOLLOWER]: Advancing to term 2
I20260812 00:59:10.415021 2653 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: false } }
I20260812 00:59:10.415102 2652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: false } }
I20260812 00:59:10.415892 2652 leader_election.cc:288] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [CANDIDATE]: Term 2 election: Requested vote from peers 3ef13f6098fe4535af6f2557d272fd2b (127.21.201.126:44537), c4c07c47632b4011bfc2a2bb1ed4a100 (127.21.201.125:39259)
I20260812 00:59:10.416524 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:10.416021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.0.0.1:55798)
0812 00:59:10.416177 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:10.416297 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260812 00:59:10.416350 2653 leader_election.cc:288] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [CANDIDATE]: Term 2 election: Requested vote from peers 3ef13f6098fe4535af6f2557d272fd2b (127.21.201.126:44537), 2e5afbe350a9486e800c519579288778 (127.21.201.124:39009)
I20260812 00:59:10.417296 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:10.416843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.0.0.1:55812)
0812 00:59:10.417031 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:10.417125 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260812 00:59:10.417399 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:10.417054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.0.0.1:55814)
0812 00:59:10.417200 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:10.417285 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260812 00:59:10.417640 1318 leader_election.cc:334] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3ef13f6098fe4535af6f2557d272fd2b (127.21.201.126:44537): Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
I20260812 00:59:10.417928 1703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2e5afbe350a9486e800c519579288778"
I20260812 00:59:10.417932 1364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e5afbe350a9486e800c519579288778" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100"
I20260812 00:59:10.418205 1703 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c4c07c47632b4011bfc2a2bb1ed4a100 in current term 2: Already voted for candidate 2e5afbe350a9486e800c519579288778 in this term.
I20260812 00:59:10.418218 1364 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2e5afbe350a9486e800c519579288778 in current term 2: Already voted for candidate c4c07c47632b4011bfc2a2bb1ed4a100 in this term.
I20260812 00:59:10.419224 1316 leader_election.cc:302] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4c07c47632b4011bfc2a2bb1ed4a100; no voters: 2e5afbe350a9486e800c519579288778, 3ef13f6098fe4535af6f2557d272fd2b
I20260812 00:59:10.419575 2653 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 00:59:10.419618 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:10.419153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.0.0.1:55830)
0812 00:59:10.419359 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:10.419464 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260812 00:59:10.419845 1666 leader_election.cc:334] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3ef13f6098fe4535af6f2557d272fd2b (127.21.201.126:44537): Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
I20260812 00:59:10.420002 1666 leader_election.cc:302] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e5afbe350a9486e800c519579288778; no voters: 3ef13f6098fe4535af6f2557d272fd2b, c4c07c47632b4011bfc2a2bb1ed4a100
I20260812 00:59:10.420270 2652 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 00:59:10.877548 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:10.876087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.21.201.65:57849)
0812 00:59:10.877018 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:10.877198 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":897,"threads_started":1}
I20260812 00:59:10.878844 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:10.876853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.21.201.67:60527)
0812 00:59:10.878539 (+ 1686us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:10.878654 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1559,"thread_start_us":1440,"threads_started":1}
I20260812 00:59:10.880899 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:10.879930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.21.201.66:58907)
0812 00:59:10.880543 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:10.880654 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":400,"threads_started":1}
I20260812 00:59:11.880211 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:11.879038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.21.201.65:51705)
0812 00:59:11.879761 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:11.879918 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":388,"threads_started":1}
I20260812 00:59:11.883332 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:11.879880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.21.201.67:54895)
0812 00:59:11.882963 (+ 3083us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:11.883085 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2953,"thread_start_us":2826,"threads_started":1}
I20260812 00:59:11.885865 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:11.883983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.21.201.66:38269)
0812 00:59:11.885426 (+ 1443us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:11.885601 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1263,"thread_start_us":1073,"threads_started":1}
I20260812 00:59:12.047536 2683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:59:12.047920 2683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: false } }
I20260812 00:59:12.048874 2683 leader_election.cc:288] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3ef13f6098fe4535af6f2557d272fd2b (127.21.201.126:44537), c4c07c47632b4011bfc2a2bb1ed4a100 (127.21.201.125:39259)
I20260812 00:59:12.050285 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.049194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.0.0.1:56076)
0812 00:59:12.049825 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:12.049993 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":309,"threads_started":1}
I20260812 00:59:12.050379 1364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e5afbe350a9486e800c519579288778" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" is_pre_election: true
I20260812 00:59:12.050720 1364 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e5afbe350a9486e800c519579288778 in term 2.
I20260812 00:59:12.051559 1666 leader_election.cc:302] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e5afbe350a9486e800c519579288778, c4c07c47632b4011bfc2a2bb1ed4a100; no voters:
I20260812 00:59:12.051894 2683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 00:59:12.052034 2683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:59:12.052148 2683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 2 FOLLOWER]: Advancing to term 3
I20260812 00:59:12.052539 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.052038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.0.0.1:56080)
0812 00:59:12.052236 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:12.052359 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 00:59:12.052757 1666 leader_election.cc:334] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3ef13f6098fe4535af6f2557d272fd2b (127.21.201.126:44537): Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
I20260812 00:59:12.054594 2683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: false } }
I20260812 00:59:12.055230 2683 leader_election.cc:288] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [CANDIDATE]: Term 3 election: Requested vote from peers 3ef13f6098fe4535af6f2557d272fd2b (127.21.201.126:44537), c4c07c47632b4011bfc2a2bb1ed4a100 (127.21.201.125:39259)
I20260812 00:59:12.055732 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.055345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.0.0.1:56094)
0812 00:59:12.055497 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:12.055585 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260812 00:59:12.056032 1364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e5afbe350a9486e800c519579288778" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100"
I20260812 00:59:12.056300 1364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 2 FOLLOWER]: Advancing to term 3
I20260812 00:59:12.056689 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.056366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.0.0.1:56100)
0812 00:59:12.056517 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:12.056595 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 00:59:12.056849 1666 leader_election.cc:334] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3ef13f6098fe4535af6f2557d272fd2b (127.21.201.126:44537): Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
I20260812 00:59:12.058671 1364 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e5afbe350a9486e800c519579288778 in term 3.
I20260812 00:59:12.059195 1666 leader_election.cc:302] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e5afbe350a9486e800c519579288778, c4c07c47632b4011bfc2a2bb1ed4a100; no voters: 3ef13f6098fe4535af6f2557d272fd2b
I20260812 00:59:12.059566 2683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 00:59:12.060117 2683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [term 3 LEADER]: Becoming Leader. State: Replica: 2e5afbe350a9486e800c519579288778, State: Running, Role: LEADER
I20260812 00:59:12.060564 2683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: false } }
I20260812 00:59:12.062664 2689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e5afbe350a9486e800c519579288778. Latest consensus state: current_term: 3 leader_uuid: "2e5afbe350a9486e800c519579288778" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: false } } }
I20260812 00:59:12.063067 2689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [sys.catalog]: This master's current role is: LEADER
I20260812 00:59:12.064080 2691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:59:12.068992 2691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:59:12.069916 2691 catalog_manager.cc:1259] Loaded cluster ID: d4fe47a674c540db8f1ec527af6061d8
I20260812 00:59:12.070000 2691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:59:12.071272 2691 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:59:12.072253 2691 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778: Loaded TSK: 0
I20260812 00:59:12.073248 2691 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:59:12.088176 1687 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:59:12.095870 1687 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:59:12.100490 1364 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [term 3 FOLLOWER]: Refusing update from remote peer 2e5afbe350a9486e800c519579288778: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260812 00:59:12.100529 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.100056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.0.0.1:56116)
0812 00:59:12.100281 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:12.100369 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260812 00:59:12.101272 2689 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260812 00:59:12.101660 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.101300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.0.0.1:56120)
0812 00:59:12.101480 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:12.101557 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260812 00:59:12.101910 1666 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 -> Peer 3ef13f6098fe4535af6f2557d272fd2b (127.21.201.126:44537): Couldn't send request to peer 3ef13f6098fe4535af6f2557d272fd2b. Status: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 00:59:12.104365 2693 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e5afbe350a9486e800c519579288778. Latest consensus state: current_term: 3 leader_uuid: "2e5afbe350a9486e800c519579288778" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: false } } }
I20260812 00:59:12.104663 2693 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:59:12.106312 2694 mvcc.cc:205] Tried to move back new op lower bound from 7317489058196369408 to 7317489058043883520. Current Snapshot: MvccSnapshot[applied={T|T < 7317489058196369408}]
I20260812 00:59:12.106407 2689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "2e5afbe350a9486e800c519579288778" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: false } } }
I20260812 00:59:12.106735 2689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778 [sys.catalog]: This master's current role is: LEADER
I20260812 00:59:12.107496 2693 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "2e5afbe350a9486e800c519579288778" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3ef13f6098fe4535af6f2557d272fd2b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44537 } } peers { permanent_uuid: "c4c07c47632b4011bfc2a2bb1ed4a100" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 39259 } attrs { promote: false } } peers { permanent_uuid: "2e5afbe350a9486e800c519579288778" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 39009 } attrs { promote: false } } }
I20260812 00:59:12.107793 2693 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4c07c47632b4011bfc2a2bb1ed4a100 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:59:12.130148 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.118519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58436 (local address 127.21.201.65:37425)
0812 00:59:12.119923 (+ 1404us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:12.119929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:12.119950 (+ 21us) server_negotiation.cc:407] Connection header received
0812 00:59:12.120034 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:12.120038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:12.120124 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:12.121053 (+ 929us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:12.121366 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.122772 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.126475 (+ 3703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.128164 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.128380 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:12.128519 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:12.129579 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0812 00:59:12.129725 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1260,"thread_start_us":1085,"threads_started":1}
I20260812 00:59:12.130846 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.118350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:37425 (local address 127.0.0.1:58436)
0812 00:59:12.118615 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:12.118637 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:12.118769 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:12.120348 (+ 1579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:12.120354 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:12.120385 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:12.120869 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:12.120885 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.122973 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.122983 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:12.126199 (+ 3216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:12.126216 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.128773 (+ 2557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.128779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:12.129068 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:12.129106 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 00:59:12.129223 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 00:59:12.129402 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":1,"spinlock_wait_cycles":7936}
I20260812 00:59:12.131856 1883 tablet_service.cc:1467] Processing CreateTablet for tablet 7993f0a42e1944d79ec523bdc7bf6793 (DEFAULT_TABLE table=test-table [id=c3f7a55e1932493880ff5db205e8a670]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:59:12.132460 1883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7993f0a42e1944d79ec523bdc7bf6793. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:12.133746 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.119114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:42819 (local address 127.0.0.1:35754)
0812 00:59:12.120354 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:12.120381 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:12.120550 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:12.125564 (+ 5014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:12.125569 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:12.125598 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:12.126448 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:12.126462 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.128116 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.128121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:12.130731 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:12.130747 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.132889 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.132896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:12.133063 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:12.133096 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 00:59:12.133202 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 00:59:12.133360 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1050,"mutex_wait_us":45,"thread_start_us":6855,"threads_started":1}
I20260812 00:59:12.134109 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.119198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35754 (local address 127.21.201.66:42819)
0812 00:59:12.125222 (+ 6024us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:12.125228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:12.125248 (+ 20us) server_negotiation.cc:407] Connection header received
0812 00:59:12.125339 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:12.125343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:12.125405 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:12.125542 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:12.126672 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.127915 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.130935 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.132433 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.132628 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:12.132754 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:12.133292 (+ 538us) server_negotiation.cc:299] Negotiation successful
0812 00:59:12.133454 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5870,"thread_start_us":5983,"threads_started":1}
I20260812 00:59:12.135113 2108 tablet_service.cc:1467] Processing CreateTablet for tablet 7993f0a42e1944d79ec523bdc7bf6793 (DEFAULT_TABLE table=test-table [id=c3f7a55e1932493880ff5db205e8a670]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:59:12.135609 2108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7993f0a42e1944d79ec523bdc7bf6793. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:12.135975 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.119634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:36707 (local address 127.0.0.1:58348)
0812 00:59:12.126335 (+ 6701us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:12.126361 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:12.126559 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:12.127083 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:12.127088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:12.127115 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:12.127459 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:12.127470 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.129273 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.129306 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:12.131942 (+ 2636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:12.131960 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.135132 (+ 3172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.135136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:12.135256 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:12.135277 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 00:59:12.135407 (+ 130us) client_negotiation.cc:240] Negotiation successful
0812 00:59:12.135554 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6542,"mutex_wait_us":1,"thread_start_us":616,"threads_started":1}
I20260812 00:59:12.135980 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.120076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58348 (local address 127.21.201.67:36707)
0812 00:59:12.124800 (+ 4724us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:12.124806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:12.126449 (+ 1643us) server_negotiation.cc:407] Connection header received
0812 00:59:12.126731 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:12.126737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:12.126877 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:12.126993 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:12.127842 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.129151 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.132187 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.134794 (+ 2607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.134891 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:12.134974 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:12.135442 (+ 468us) server_negotiation.cc:299] Negotiation successful
0812 00:59:12.135588 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4558,"thread_start_us":5024,"threads_started":1}
I20260812 00:59:12.140340 2714 tablet_bootstrap.cc:492] T 7993f0a42e1944d79ec523bdc7bf6793 P 2ac902d212d8411bb6adfa954008ed8e: Bootstrap starting.
I20260812 00:59:12.140259 2370 tablet_service.cc:1467] Processing CreateTablet for tablet 7993f0a42e1944d79ec523bdc7bf6793 (DEFAULT_TABLE table=test-table [id=c3f7a55e1932493880ff5db205e8a670]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:59:12.140740 2370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7993f0a42e1944d79ec523bdc7bf6793. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:12.141993 2714 tablet_bootstrap.cc:654] T 7993f0a42e1944d79ec523bdc7bf6793 P 2ac902d212d8411bb6adfa954008ed8e: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:12.144276 2714 tablet_bootstrap.cc:492] T 7993f0a42e1944d79ec523bdc7bf6793 P 2ac902d212d8411bb6adfa954008ed8e: No bootstrap required, opened a new log
I20260812 00:59:12.144438 2714 ts_tablet_manager.cc:1397] T 7993f0a42e1944d79ec523bdc7bf6793 P 2ac902d212d8411bb6adfa954008ed8e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260812 00:59:12.145283 2714 raft_consensus.cc:348] T 7993f0a42e1944d79ec523bdc7bf6793 P 2ac902d212d8411bb6adfa954008ed8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac902d212d8411bb6adfa954008ed8e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 37425 } } peers { permanent_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42819 } } peers { permanent_uuid: "3a45e74843e94221ad9517a04509d375" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 36707 } }
I20260812 00:59:12.145587 2714 raft_consensus.cc:374] T 7993f0a42e1944d79ec523bdc7bf6793 P 2ac902d212d8411bb6adfa954008ed8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:12.145695 2714 raft_consensus.cc:729] T 7993f0a42e1944d79ec523bdc7bf6793 P 2ac902d212d8411bb6adfa954008ed8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac902d212d8411bb6adfa954008ed8e, State: Initialized, Role: FOLLOWER
I20260812 00:59:12.145942 2714 consensus_queue.cc:260] T 7993f0a42e1944d79ec523bdc7bf6793 P 2ac902d212d8411bb6adfa954008ed8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac902d212d8411bb6adfa954008ed8e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 37425 } } peers { permanent_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42819 } } peers { permanent_uuid: "3a45e74843e94221ad9517a04509d375" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 36707 } }
I20260812 00:59:12.146670 2717 tablet_bootstrap.cc:492] T 7993f0a42e1944d79ec523bdc7bf6793 P 3a45e74843e94221ad9517a04509d375: Bootstrap starting.
I20260812 00:59:12.147583 2720 tablet_bootstrap.cc:492] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828: Bootstrap starting.
I20260812 00:59:12.148905 2717 tablet_bootstrap.cc:654] T 7993f0a42e1944d79ec523bdc7bf6793 P 3a45e74843e94221ad9517a04509d375: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:12.149611 2720 tablet_bootstrap.cc:654] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:12.151599 2717 tablet_bootstrap.cc:492] T 7993f0a42e1944d79ec523bdc7bf6793 P 3a45e74843e94221ad9517a04509d375: No bootstrap required, opened a new log
I20260812 00:59:12.151791 2717 ts_tablet_manager.cc:1397] T 7993f0a42e1944d79ec523bdc7bf6793 P 3a45e74843e94221ad9517a04509d375: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 00:59:12.152516 2714 ts_tablet_manager.cc:1428] T 7993f0a42e1944d79ec523bdc7bf6793 P 2ac902d212d8411bb6adfa954008ed8e: Time spent starting tablet: real 0.008s user 0.001s sys 0.008s
I20260812 00:59:12.152684 2717 raft_consensus.cc:348] T 7993f0a42e1944d79ec523bdc7bf6793 P 3a45e74843e94221ad9517a04509d375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac902d212d8411bb6adfa954008ed8e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 37425 } } peers { permanent_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42819 } } peers { permanent_uuid: "3a45e74843e94221ad9517a04509d375" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 36707 } }
I20260812 00:59:12.152961 2717 raft_consensus.cc:374] T 7993f0a42e1944d79ec523bdc7bf6793 P 3a45e74843e94221ad9517a04509d375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:12.153062 2717 raft_consensus.cc:729] T 7993f0a42e1944d79ec523bdc7bf6793 P 3a45e74843e94221ad9517a04509d375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a45e74843e94221ad9517a04509d375, State: Initialized, Role: FOLLOWER
I20260812 00:59:12.153339 2717 consensus_queue.cc:260] T 7993f0a42e1944d79ec523bdc7bf6793 P 3a45e74843e94221ad9517a04509d375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac902d212d8411bb6adfa954008ed8e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 37425 } } peers { permanent_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42819 } } peers { permanent_uuid: "3a45e74843e94221ad9517a04509d375" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 36707 } }
I20260812 00:59:12.154874 2717 ts_tablet_manager.cc:1428] T 7993f0a42e1944d79ec523bdc7bf6793 P 3a45e74843e94221ad9517a04509d375: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:59:12.155959 2720 tablet_bootstrap.cc:492] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828: No bootstrap required, opened a new log
I20260812 00:59:12.156132 2720 ts_tablet_manager.cc:1397] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20260812 00:59:12.156674 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.156050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.21.201.67:41493)
0812 00:59:12.156388 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:12.156502 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207}
I20260812 00:59:12.157411 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.156429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.21.201.65:48307)
0812 00:59:12.157148 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:12.157247 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":597}
I20260812 00:59:12.157442 2720 raft_consensus.cc:348] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac902d212d8411bb6adfa954008ed8e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 37425 } } peers { permanent_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42819 } } peers { permanent_uuid: "3a45e74843e94221ad9517a04509d375" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 36707 } }
I20260812 00:59:12.157720 2720 raft_consensus.cc:374] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:12.157846 2720 raft_consensus.cc:729] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f446bf9ca014a99b3ddcb2f0e2bc828, State: Initialized, Role: FOLLOWER
I20260812 00:59:12.158201 2720 consensus_queue.cc:260] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac902d212d8411bb6adfa954008ed8e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 37425 } } peers { permanent_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42819 } } peers { permanent_uuid: "3a45e74843e94221ad9517a04509d375" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 36707 } }
I20260812 00:59:12.159186 2720 ts_tablet_manager.cc:1428] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:59:12.160509 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.160057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.21.201.66:33135)
0812 00:59:12.160249 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:12.160357 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 00:59:12.161319 2227 heartbeater.cc:503] Master 127.21.201.124:39009 was elected leader, sending a full tablet report...
I20260812 00:59:12.161990 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.161388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.21.201.66:55085)
0812 00:59:12.161713 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:12.161803 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218}
I20260812 00:59:12.349337 2730 raft_consensus.cc:482] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:59:12.349604 2730 raft_consensus.cc:504] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac902d212d8411bb6adfa954008ed8e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 37425 } } peers { permanent_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42819 } } peers { permanent_uuid: "3a45e74843e94221ad9517a04509d375" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 36707 } }
I20260812 00:59:12.351543 2730 leader_election.cc:288] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ac902d212d8411bb6adfa954008ed8e (127.21.201.65:37425), 3a45e74843e94221ad9517a04509d375 (127.21.201.67:36707)
I20260812 00:59:12.358911 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.352225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:55789 (local address 127.21.201.67:36707)
0812 00:59:12.352483 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:12.352507 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:12.353057 (+ 550us) server_negotiation.cc:407] Connection header received
0812 00:59:12.353236 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:12.353242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:12.353326 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:12.353431 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:12.354041 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.354873 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.356772 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.357567 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.357790 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:12.357886 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:12.358450 (+ 564us) server_negotiation.cc:299] Negotiation successful
0812 00:59:12.358600 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260812 00:59:12.359099 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.351961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:60087 (local address 127.21.201.65:37425)
0812 00:59:12.352332 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:12.352339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:12.352364 (+ 25us) server_negotiation.cc:407] Connection header received
0812 00:59:12.352574 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:12.352583 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:12.352679 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:12.352782 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:12.353485 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.354429 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.356669 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.357674 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.357836 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:12.357926 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:12.358407 (+ 481us) server_negotiation.cc:299] Negotiation successful
0812 00:59:12.358562 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 00:59:12.359375 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.351859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:37425 (local address 127.21.201.66:60087)
0812 00:59:12.352166 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:12.352192 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:12.352351 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:12.352854 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:12.352858 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:12.352883 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:12.353291 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:12.353306 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.354587 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.354594 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:12.356487 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:12.356501 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.358073 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.358078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:12.358239 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:12.358262 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 00:59:12.358380 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 00:59:12.358606 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":169,"spinlock_wait_cycles":11008}
I20260812 00:59:12.359448 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.351913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:36707 (local address 127.21.201.66:55789)
0812 00:59:12.352937 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:12.352962 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:12.353079 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:12.353480 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:12.353484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:12.353510 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:12.353874 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:12.353886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.355034 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.355038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:12.356597 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:12.356618 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.357800 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.357808 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:12.358004 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:12.358037 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 00:59:12.358399 (+ 362us) client_negotiation.cc:240] Negotiation successful
0812 00:59:12.358586 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"mutex_wait_us":6,"spinlock_wait_cycles":2944,"thread_start_us":585,"threads_started":1}
I20260812 00:59:12.360189 1915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f0a42e1944d79ec523bdc7bf6793" candidate_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac902d212d8411bb6adfa954008ed8e" is_pre_election: true
I20260812 00:59:12.360306 2402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f0a42e1944d79ec523bdc7bf6793" candidate_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a45e74843e94221ad9517a04509d375" is_pre_election: true
I20260812 00:59:12.360569 2402 raft_consensus.cc:2454] T 7993f0a42e1944d79ec523bdc7bf6793 P 3a45e74843e94221ad9517a04509d375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f446bf9ca014a99b3ddcb2f0e2bc828 in term 0.
I20260812 00:59:12.360571 1915 raft_consensus.cc:2454] T 7993f0a42e1944d79ec523bdc7bf6793 P 2ac902d212d8411bb6adfa954008ed8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f446bf9ca014a99b3ddcb2f0e2bc828 in term 0.
I20260812 00:59:12.361197 2043 leader_election.cc:302] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2ac902d212d8411bb6adfa954008ed8e, 3a45e74843e94221ad9517a04509d375, 6f446bf9ca014a99b3ddcb2f0e2bc828; no voters:
I20260812 00:59:12.361673 2730 raft_consensus.cc:2781] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:59:12.361829 2730 raft_consensus.cc:482] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:59:12.361939 2730 raft_consensus.cc:3037] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:12.364585 2730 raft_consensus.cc:504] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac902d212d8411bb6adfa954008ed8e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 37425 } } peers { permanent_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42819 } } peers { permanent_uuid: "3a45e74843e94221ad9517a04509d375" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 36707 } }
I20260812 00:59:12.365213 2730 leader_election.cc:288] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [CANDIDATE]: Term 1 election: Requested vote from peers 2ac902d212d8411bb6adfa954008ed8e (127.21.201.65:37425), 3a45e74843e94221ad9517a04509d375 (127.21.201.67:36707)
I20260812 00:59:12.365839 1915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f0a42e1944d79ec523bdc7bf6793" candidate_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac902d212d8411bb6adfa954008ed8e"
I20260812 00:59:12.365854 2402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7993f0a42e1944d79ec523bdc7bf6793" candidate_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a45e74843e94221ad9517a04509d375"
I20260812 00:59:12.366109 2402 raft_consensus.cc:3037] T 7993f0a42e1944d79ec523bdc7bf6793 P 3a45e74843e94221ad9517a04509d375 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:12.366134 1915 raft_consensus.cc:3037] T 7993f0a42e1944d79ec523bdc7bf6793 P 2ac902d212d8411bb6adfa954008ed8e [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:12.369388 1915 raft_consensus.cc:2454] T 7993f0a42e1944d79ec523bdc7bf6793 P 2ac902d212d8411bb6adfa954008ed8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f446bf9ca014a99b3ddcb2f0e2bc828 in term 1.
I20260812 00:59:12.369412 2402 raft_consensus.cc:2454] T 7993f0a42e1944d79ec523bdc7bf6793 P 3a45e74843e94221ad9517a04509d375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f446bf9ca014a99b3ddcb2f0e2bc828 in term 1.
I20260812 00:59:12.369904 2043 leader_election.cc:302] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ac902d212d8411bb6adfa954008ed8e, 6f446bf9ca014a99b3ddcb2f0e2bc828; no voters:
I20260812 00:59:12.370290 2730 raft_consensus.cc:2781] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:59:12.370819 2730 raft_consensus.cc:686] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [term 1 LEADER]: Becoming Leader. State: Replica: 6f446bf9ca014a99b3ddcb2f0e2bc828, State: Running, Role: LEADER
I20260812 00:59:12.371299 2730 consensus_queue.cc:237] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac902d212d8411bb6adfa954008ed8e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 37425 } } peers { permanent_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42819 } } peers { permanent_uuid: "3a45e74843e94221ad9517a04509d375" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 36707 } }
I20260812 00:59:12.372087 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.371617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.21.201.66:33297)
0812 00:59:12.371841 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:12.371928 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260812 00:59:12.376158 1685 catalog_manager.cc:5495] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f446bf9ca014a99b3ddcb2f0e2bc828 (127.21.201.66). New cstate: current_term: 1 leader_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac902d212d8411bb6adfa954008ed8e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 37425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f446bf9ca014a99b3ddcb2f0e2bc828" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a45e74843e94221ad9517a04509d375" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 36707 } health_report { overall_health: UNKNOWN } } }
I20260812 00:59:12.435647 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.429381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35768 (local address 127.21.201.66:42819)
0812 00:59:12.429581 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:12.429590 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:12.429635 (+ 45us) server_negotiation.cc:407] Connection header received
0812 00:59:12.429956 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:12.429960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:12.430054 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:12.430232 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:59:12.430671 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.431490 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.433134 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.433723 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.433869 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:12.434367 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:59:12.434963 (+ 596us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:59:12.435153 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:12.435255 (+ 102us) server_negotiation.cc:299] Negotiation successful
0812 00:59:12.435345 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 00:59:12.441506 2402 raft_consensus.cc:1261] T 7993f0a42e1944d79ec523bdc7bf6793 P 3a45e74843e94221ad9517a04509d375 [term 1 FOLLOWER]: Refusing update from remote peer 6f446bf9ca014a99b3ddcb2f0e2bc828: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:59:12.442476 2734 consensus_queue.cc:1035] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a45e74843e94221ad9517a04509d375" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 36707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:59:12.442909 1915 raft_consensus.cc:1261] T 7993f0a42e1944d79ec523bdc7bf6793 P 2ac902d212d8411bb6adfa954008ed8e [term 1 FOLLOWER]: Refusing update from remote peer 6f446bf9ca014a99b3ddcb2f0e2bc828: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:59:12.443783 2734 consensus_queue.cc:1035] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac902d212d8411bb6adfa954008ed8e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 37425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:59:12.444482 2494 heartbeater.cc:503] Master 127.21.201.124:39009 was elected leader, sending a full tablet report...
I20260812 00:59:12.445995 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.445333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.21.201.67:60893)
0812 00:59:12.445704 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:12.445802 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253}
I20260812 00:59:12.446374 1985 heartbeater.cc:503] Master 127.21.201.124:39009 was elected leader, sending a full tablet report...
I20260812 00:59:12.447315 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.446697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.21.201.65:39753)
0812 00:59:12.447007 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:12.447110 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260812 00:59:12.457332 2737 mvcc.cc:205] Tried to move back new op lower bound from 7317489059590823936 to 7317489059317272576. Current Snapshot: MvccSnapshot[applied={T|T < 7317489059590823936}]
I20260812 00:59:12.459270 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.458337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.21.201.65:41679)
0812 00:59:12.458664 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:12.458759 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216}
I20260812 00:59:12.461735 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.461072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.21.201.66:57535)
0812 00:59:12.461434 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:12.461566 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247}
I20260812 00:59:12.467278 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.466718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.21.201.66:58785)
0812 00:59:12.466967 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:12.467088 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260812 00:59:12.471975 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.471070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.21.201.67:58565)
0812 00:59:12.471399 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:12.471549 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260812 00:59:12.481789 1687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:59:12.482352 1687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:59:12.490387 1687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e5afbe350a9486e800c519579288778: Sending DeleteTablet for 3 replicas of tablet 7993f0a42e1944d79ec523bdc7bf6793
I20260812 00:59:12.491957 2108 tablet_service.cc:1514] Processing DeleteTablet for tablet 7993f0a42e1944d79ec523bdc7bf6793 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:59:12 UTC) from {username='oryx', principal='oryx/127.21.201.124@KRBTEST.COM'} at 127.0.0.1:35754
I20260812 00:59:12.492079 2370 tablet_service.cc:1514] Processing DeleteTablet for tablet 7993f0a42e1944d79ec523bdc7bf6793 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:59:12 UTC) from {username='oryx', principal='oryx/127.21.201.124@KRBTEST.COM'} at 127.0.0.1:58348
I20260812 00:59:12.492563 1883 tablet_service.cc:1514] Processing DeleteTablet for tablet 7993f0a42e1944d79ec523bdc7bf6793 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:59:12 UTC) from {username='oryx', principal='oryx/127.21.201.124@KRBTEST.COM'} at 127.0.0.1:58436
I20260812 00:59:12.493534 2756 tablet_replica.cc:330] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828: stopping tablet replica
I20260812 00:59:12.493743 2756 raft_consensus.cc:2229] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:59:12.494465 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 1657
I20260812 00:59:12.494450 2756 raft_consensus.cc:2258] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:59:12.506381 2758 tablet_replica.cc:330] T 7993f0a42e1944d79ec523bdc7bf6793 P 2ac902d212d8411bb6adfa954008ed8e: stopping tablet replica
I20260812 00:59:12.506659 2758 raft_consensus.cc:2229] T 7993f0a42e1944d79ec523bdc7bf6793 P 2ac902d212d8411bb6adfa954008ed8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:59:12.507119 2758 raft_consensus.cc:2258] T 7993f0a42e1944d79ec523bdc7bf6793 P 2ac902d212d8411bb6adfa954008ed8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:59:12.508050 2756 ts_tablet_manager.cc:1905] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:59:12.508639 2758 ts_tablet_manager.cc:1905] T 7993f0a42e1944d79ec523bdc7bf6793 P 2ac902d212d8411bb6adfa954008ed8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:59:12.516337 2757 tablet_replica.cc:330] T 7993f0a42e1944d79ec523bdc7bf6793 P 3a45e74843e94221ad9517a04509d375: stopping tablet replica
I20260812 00:59:12.516647 2757 raft_consensus.cc:2229] T 7993f0a42e1944d79ec523bdc7bf6793 P 3a45e74843e94221ad9517a04509d375 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:59:12.517046 2757 raft_consensus.cc:2258] T 7993f0a42e1944d79ec523bdc7bf6793 P 3a45e74843e94221ad9517a04509d375 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:59:12.518216 2758 ts_tablet_manager.cc:1918] T 7993f0a42e1944d79ec523bdc7bf6793 P 2ac902d212d8411bb6adfa954008ed8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:59:12.518411 2758 log.cc:1196] T 7993f0a42e1944d79ec523bdc7bf6793 P 2ac902d212d8411bb6adfa954008ed8e: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-0/wal/wals/7993f0a42e1944d79ec523bdc7bf6793
I20260812 00:59:12.518905 2757 ts_tablet_manager.cc:1905] T 7993f0a42e1944d79ec523bdc7bf6793 P 3a45e74843e94221ad9517a04509d375: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:59:12.518976 2758 ts_tablet_manager.cc:1939] T 7993f0a42e1944d79ec523bdc7bf6793 P 2ac902d212d8411bb6adfa954008ed8e: Deleting consensus metadata
I20260812 00:59:12.520466 1668 catalog_manager.cc:4841] TS 2ac902d212d8411bb6adfa954008ed8e (127.21.201.65:37425): tablet 7993f0a42e1944d79ec523bdc7bf6793 (table test-table [id=c3f7a55e1932493880ff5db205e8a670]) successfully deleted
I20260812 00:59:12.522423 2756 ts_tablet_manager.cc:1918] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:59:12.522598 2756 log.cc:1196] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/7993f0a42e1944d79ec523bdc7bf6793
I20260812 00:59:12.523175 2756 ts_tablet_manager.cc:1939] T 7993f0a42e1944d79ec523bdc7bf6793 P 6f446bf9ca014a99b3ddcb2f0e2bc828: Deleting consensus metadata
I20260812 00:59:12.523800 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.496289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58448 (local address 127.21.201.65:37425)
0812 00:59:12.496546 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:12.496556 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:12.506211 (+ 9655us) server_negotiation.cc:407] Connection header received
0812 00:59:12.506442 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:12.506448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:12.506554 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:12.506667 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:12.508599 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.509482 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.511217 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.511791 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.511922 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:12.515714 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:12.515731 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:12.515803 (+ 72us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:12.515862 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:12.520366 (+ 4504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:12.521315 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:12.521325 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:12.521329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:12.521530 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:12.521924 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:12.521929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:12.521933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:12.522460 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:12.522793 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:12.523128 (+ 335us) server_negotiation.cc:299] Negotiation successful
0812 00:59:12.523388 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 00:59:12.524611 1979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:12.524955 1667 catalog_manager.cc:4841] TS 6f446bf9ca014a99b3ddcb2f0e2bc828 (127.21.201.66:42819): tablet 7993f0a42e1944d79ec523bdc7bf6793 (table test-table [id=c3f7a55e1932493880ff5db205e8a670]) successfully deleted
I20260812 00:59:12.527099 2757 ts_tablet_manager.cc:1918] T 7993f0a42e1944d79ec523bdc7bf6793 P 3a45e74843e94221ad9517a04509d375: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:59:12.527338 2757 log.cc:1196] T 7993f0a42e1944d79ec523bdc7bf6793 P 3a45e74843e94221ad9517a04509d375: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/7993f0a42e1944d79ec523bdc7bf6793
I20260812 00:59:12.527983 2757 ts_tablet_manager.cc:1939] T 7993f0a42e1944d79ec523bdc7bf6793 P 3a45e74843e94221ad9517a04509d375: Deleting consensus metadata
I20260812 00:59:12.529367 1665 catalog_manager.cc:4841] TS 3a45e74843e94221ad9517a04509d375 (127.21.201.67:36707): tablet 7993f0a42e1944d79ec523bdc7bf6793 (table test-table [id=c3f7a55e1932493880ff5db205e8a670]) successfully deleted
I20260812 00:59:12.643826 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.642683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.0.0.1:56134)
0812 00:59:12.643335 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:12.643457 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":350,"threads_started":1}
I20260812 00:59:12.645721 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.645260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.0.0.1:56146)
0812 00:59:12.645460 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:12.645578 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260812 00:59:12.713377 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 1657
I20260812 00:59:12.744369 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 2009
I20260812 00:59:12.763324 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.745385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35782 (local address 127.21.201.66:42819)
0812 00:59:12.745640 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:12.745647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:12.745778 (+ 131us) server_negotiation.cc:407] Connection header received
0812 00:59:12.746019 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:12.746024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:12.746142 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:12.746269 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:12.748396 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.749554 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.751759 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.752458 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.752620 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:12.755555 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:12.755570 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:12.755591 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:12.755658 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:12.760088 (+ 4430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:12.760902 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:12.760912 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:12.760916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:12.761125 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:12.761510 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:12.761515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:12.761518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:12.762024 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:12.762347 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:12.762644 (+ 297us) server_negotiation.cc:299] Negotiation successful
0812 00:59:12.762906 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260812 00:59:12.764292 2220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:12.894919 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 2009
I20260812 00:59:12.926512 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 2250
I20260812 00:59:12.943688 2770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:12.927692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58354 (local address 127.21.201.67:36707)
0812 00:59:12.928598 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:12.928604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:12.928647 (+ 43us) server_negotiation.cc:407] Connection header received
0812 00:59:12.928729 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:12.928734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:12.928815 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:12.928945 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:12.931034 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.932197 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.933529 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:12.933951 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:12.934054 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:12.937111 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:12.937130 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:12.937148 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:12.937204 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:12.940609 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:12.941431 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:12.941439 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:12.941441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:12.941622 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:12.942128 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:12.942132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:12.942135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:12.942541 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:12.942802 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:12.943171 (+ 369us) server_negotiation.cc:299] Negotiation successful
0812 00:59:12.943369 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":504,"threads_started":1}
I20260812 00:59:12.945047 2488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:13.055930 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 2250
I20260812 00:59:13.081918 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 1109
I20260812 00:59:13.083508 1399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:13.121137 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:13.120622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.0.0.1:56162)
0812 00:59:13.120835 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:13.120948 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260812 00:59:13.122296 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:13.121913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44537 (local address 127.0.0.1:56176)
0812 00:59:13.122075 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:13.122160 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260812 00:59:13.213647 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 1109
I20260812 00:59:13.245451 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 1279
I20260812 00:59:13.246753 1740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:13.383632 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 1279
2026-08-12T00:59:13Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (12886 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:59:13 dist-test-slave-7f01 krb5kdc[2782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:59:13 dist-test-slave-7f01 krb5kdc[2782](info): set up 2 sockets
Aug 12 00:59:13 dist-test-slave-7f01 krb5kdc[2782](info): commencing operation
krb5kdc: starting...
W20260812 00:59:14.829730 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.375s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 00:59:15 dist-test-slave-7f01 krb5kdc[2782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:59:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:59:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.126.keytab.
Entry for principal oryx/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.126.keytab.
I20260812 00:59:15.133385 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:42259
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.126.keytab
--principal=oryx/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:44081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:42259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:59:15.499977 2804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:15.500330 2804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:15.500432 2804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:15.508034 2804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:59:15.508164 2804 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 00:59:15.508209 2804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:15.508275 2804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:15.508328 2804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:59:15.508399 2804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:59:15.518761 2804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:44081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:42259
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.126.keytab
--principal=oryx/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:42259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:15.520174 2804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:15.521633 2804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:15.536053 2816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:15.536053 2817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:15.536084 2819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:15.537139 2804 server_base.cc:1034] running on GCE node
I20260812 00:59:15.538007 2804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:15.539541 2804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:15.540822 2804 hybrid_clock.cc:648] HybridClock initialized: now 1786496355540746 us; error 76 us; skew 500 ppm
I20260812 00:59:15.541181 2804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:15 dist-test-slave-7f01 krb5kdc[2782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496355, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:15.544834 2804 init.cc:377] Logged in from keytab as oryx/127.21.201.126@KRBTEST.COM (short username oryx)
I20260812 00:59:15.546586 2804 webserver.cc:460] Webserver started at http://127.21.201.126:42857/ using document root <none> and password file <none>
I20260812 00:59:15.547394 2804 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:15.547502 2804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:15.547955 2804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:59:15.551223 2804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "8e09f1a6ea1e47e6b18e9fe50fe2cb8e"
format_stamp: "Formatted at 2026-08-12 00:59:15 on dist-test-slave-7f01"
server_key: "fcc3164eab59a312d660754cc515c042"
server_key_iv: "cb3c0cfed48b8aad5e9fc607be7d8420"
server_key_version: "encryptionkey@0"
I20260812 00:59:15.552157 2804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "8e09f1a6ea1e47e6b18e9fe50fe2cb8e"
format_stamp: "Formatted at 2026-08-12 00:59:15 on dist-test-slave-7f01"
server_key: "fcc3164eab59a312d660754cc515c042"
server_key_iv: "cb3c0cfed48b8aad5e9fc607be7d8420"
server_key_version: "encryptionkey@0"
I20260812 00:59:15.557750 2804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 00:59:15.561720 2829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:15.563166 2804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 00:59:15.563371 2804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "8e09f1a6ea1e47e6b18e9fe50fe2cb8e"
format_stamp: "Formatted at 2026-08-12 00:59:15 on dist-test-slave-7f01"
server_key: "fcc3164eab59a312d660754cc515c042"
server_key_iv: "cb3c0cfed48b8aad5e9fc607be7d8420"
server_key_version: "encryptionkey@0"
I20260812 00:59:15.563628 2804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:15.586459 2804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:15.591689 2804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:15.592095 2804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:15.617903 2804 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:42259
I20260812 00:59:15.617923 2940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:42259 every 8 connection(s)
I20260812 00:59:15.619438 2804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 00:59:15.622944 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 2804
I20260812 00:59:15.623176 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 00:59:15.624073 22309 external_mini_cluster.cc:1428] Setting key d6e93c6481738938fc4a5f66ef3fea68
I20260812 00:59:15.624089 2941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:15.640054 2941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e: Bootstrap starting.
Aug 12 00:59:15 dist-test-slave-7f01 krb5kdc[2782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM
I20260812 00:59:15.643520 2941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:15.644731 2941 log.cc:824] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e: Log is configured to *not* fsync() on all Append() calls
I20260812 00:59:15.647864 2941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e: No bootstrap required, opened a new log
I20260812 00:59:15.649343 2945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:15.626393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48440 (local address 127.21.201.126:42259)
0812 00:59:15.627320 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:15.627333 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:15.627379 (+ 46us) server_negotiation.cc:407] Connection header received
0812 00:59:15.629231 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:15.629257 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:15.629668 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:15.630017 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:15.631972 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:15.633340 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:15.635222 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:15.636118 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:15.636269 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:15.641319 (+ 5050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:15.641327 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:15.641343 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:15.641399 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:15.644889 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:15.645704 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:15.645721 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:15.645727 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:15.645914 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:15.646339 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:15.646344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:15.646348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:15.646902 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:15.647219 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:15.647695 (+ 476us) server_negotiation.cc:299] Negotiation successful
0812 00:59:15.648331 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":471,"threads_started":1}
I20260812 00:59:15.656627 2941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e09f1a6ea1e47e6b18e9fe50fe2cb8e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42259 } }
I20260812 00:59:15.657071 2941 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:15.657191 2941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e09f1a6ea1e47e6b18e9fe50fe2cb8e, State: Initialized, Role: FOLLOWER
I20260812 00:59:15.657771 2941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e09f1a6ea1e47e6b18e9fe50fe2cb8e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42259 } }
I20260812 00:59:15.658048 2941 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:59:15.658205 2941 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:59:15.658389 2941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:15.661334 2941 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e09f1a6ea1e47e6b18e9fe50fe2cb8e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42259 } }
I20260812 00:59:15.661906 2941 leader_election.cc:302] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e09f1a6ea1e47e6b18e9fe50fe2cb8e; no voters:
I20260812 00:59:15.662520 2941 leader_election.cc:288] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:59:15.662681 2948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:59:15.663488 2948 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e [term 1 LEADER]: Becoming Leader. State: Replica: 8e09f1a6ea1e47e6b18e9fe50fe2cb8e, State: Running, Role: LEADER
I20260812 00:59:15.664147 2948 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e09f1a6ea1e47e6b18e9fe50fe2cb8e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42259 } }
I20260812 00:59:15.664922 2941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:59:15.667629 2950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e09f1a6ea1e47e6b18e9fe50fe2cb8e. Latest consensus state: current_term: 1 leader_uuid: "8e09f1a6ea1e47e6b18e9fe50fe2cb8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e09f1a6ea1e47e6b18e9fe50fe2cb8e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42259 } } }
I20260812 00:59:15.668170 2950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e [sys.catalog]: This master's current role is: LEADER
I20260812 00:59:15.668433 2949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e09f1a6ea1e47e6b18e9fe50fe2cb8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e09f1a6ea1e47e6b18e9fe50fe2cb8e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42259 } } }
I20260812 00:59:15.668780 2949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e [sys.catalog]: This master's current role is: LEADER
I20260812 00:59:15.670558 2968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:59:15.676399 2968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:59:15.686376 2968 catalog_manager.cc:1347] Generated new cluster ID: 0be1681cb66f48888cb06a0fd5f34e58
I20260812 00:59:15.686528 2968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:59:15.698648 2968 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:59:15.700896 2968 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:59:15.713244 2968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e: Generated new TSK 0
I20260812 00:59:15.714463 2968 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.201.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.201.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.201.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.65.keytab.
Entry for principal oryx/127.21.201.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.65.keytab.
I20260812 00:59:15.789794 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:0
--local_ip_for_outbound_sockets=127.21.201.65
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:42259
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.65.keytab
--principal=oryx/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:44081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:59:16.165017 2988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:16.165338 2988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:16.165447 2988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:16.173909 2988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:16.174033 2988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:16.174204 2988 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:59:16.174338 2988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 00:59:16.186630 2988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:44081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.65.keytab
--principal=oryx/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.201.126:42259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:16.188223 2988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:16.189632 2988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:16.203986 2994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:16.203999 2998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:16.204753 3001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:16.204998 2988 server_base.cc:1034] running on GCE node
I20260812 00:59:16.205699 2988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:16.206691 2988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:16.207993 2988 hybrid_clock.cc:648] HybridClock initialized: now 1786496356207906 us; error 56 us; skew 500 ppm
I20260812 00:59:16.208334 2988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:16 dist-test-slave-7f01 krb5kdc[2782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496356, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:16.211808 2988 init.cc:377] Logged in from keytab as oryx/127.21.201.65@KRBTEST.COM (short username oryx)
I20260812 00:59:16.213604 2988 webserver.cc:460] Webserver started at http://127.21.201.65:46159/ using document root <none> and password file <none>
I20260812 00:59:16.214365 2988 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:16.214512 2988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:16.214864 2988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:59:16.218099 2988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-0/data/instance:
uuid: "8724c11a48bb4a7182aa973cd95bdb92"
format_stamp: "Formatted at 2026-08-12 00:59:16 on dist-test-slave-7f01"
server_key: "3ecd14f1b0ed53842473493266cdab82"
server_key_iv: "a07edfb5e685a1580a1b2f25ee658f28"
server_key_version: "encryptionkey@0"
I20260812 00:59:16.219053 2988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance:
uuid: "8724c11a48bb4a7182aa973cd95bdb92"
format_stamp: "Formatted at 2026-08-12 00:59:16 on dist-test-slave-7f01"
server_key: "3ecd14f1b0ed53842473493266cdab82"
server_key_iv: "a07edfb5e685a1580a1b2f25ee658f28"
server_key_version: "encryptionkey@0"
I20260812 00:59:16.225000 2988 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 00:59:16.229354 3012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:16.230921 2988 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260812 00:59:16.231178 2988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "8724c11a48bb4a7182aa973cd95bdb92"
format_stamp: "Formatted at 2026-08-12 00:59:16 on dist-test-slave-7f01"
server_key: "3ecd14f1b0ed53842473493266cdab82"
server_key_iv: "a07edfb5e685a1580a1b2f25ee658f28"
server_key_version: "encryptionkey@0"
I20260812 00:59:16.231453 2988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:16.255026 2988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:16.259928 2988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:16.260645 2988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:16.265015 2988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:59:16.265230 2988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260812 00:59:16.265486 2988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:59:16.265571 2988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 00:59:16 dist-test-slave-7f01 krb5kdc[2782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496356, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM
I20260812 00:59:16.294523 3023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:16.265108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:34465 (local address 127.21.201.126:42259)
0812 00:59:16.265646 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:16.265652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:16.267315 (+ 1663us) server_negotiation.cc:407] Connection header received
0812 00:59:16.270805 (+ 3490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:16.270821 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:16.271001 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:16.271226 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:16.274686 (+ 3460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:16.276325 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:16.278229 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:16.278989 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:16.279147 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:16.285731 (+ 6584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:16.285740 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:16.285750 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:16.285796 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:16.289254 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:16.290681 (+ 1427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:16.290689 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:16.290692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:16.290961 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:16.291809 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:16.291816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:16.291820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:16.292261 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:16.292461 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:16.293879 (+ 1418us) server_negotiation.cc:299] Negotiation successful
0812 00:59:16.294180 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":273,"threads_started":1}
I20260812 00:59:16.296526 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:16.265973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:42259 (local address 127.21.201.65:34465)
0812 00:59:16.267092 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:16.267154 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:16.270238 (+ 3084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:16.271895 (+ 1657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:16.271909 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:16.273296 (+ 1387us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:16.274354 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:16.274403 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:16.276560 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:16.276564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:16.278054 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:16.278069 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:16.279318 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:16.279332 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:16.279730 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:16.281112 (+ 1382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:16.281160 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:16.285371 (+ 4211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:16.289437 (+ 4066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:16.289453 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:16.289480 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:16.290339 (+ 859us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:16.291165 (+ 826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:16.291172 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:16.291179 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:16.291544 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:16.292444 (+ 900us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:16.292451 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:16.293150 (+ 699us) client_negotiation.cc:769] Sending connection context
0812 00:59:16.293596 (+ 446us) client_negotiation.cc:240] Negotiation successful
0812 00:59:16.294440 (+ 844us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":404,"threads_started":1}
I20260812 00:59:16.327345 3023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:16.305196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:44323 (local address 127.21.201.126:42259)
0812 00:59:16.305410 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:16.305417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:16.311253 (+ 5836us) server_negotiation.cc:407] Connection header received
0812 00:59:16.311658 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:16.311667 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:16.311835 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:16.311966 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:16.314385 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:16.315408 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:16.317660 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:16.318348 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:16.318498 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:16.321054 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:16.321059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:16.321062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:16.321097 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:16.324500 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:16.325215 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:16.325220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:16.325224 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:16.325353 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:16.325746 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:16.325755 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:16.325758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:16.326070 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:16.326236 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:16.326657 (+ 421us) server_negotiation.cc:299] Negotiation successful
0812 00:59:16.326946 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 00:59:16.327440 3131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:16.304807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:42259 (local address 127.21.201.65:44323)
0812 00:59:16.311099 (+ 6292us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:16.311127 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:16.311410 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:16.312046 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:16.312053 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:16.313290 (+ 1237us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:16.314102 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:16.314120 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:16.315635 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:16.315643 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:16.317499 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:16.317520 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:16.318579 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:16.318586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:16.318751 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:16.319543 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:16.319561 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:16.320815 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:16.324644 (+ 3829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:16.324649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:16.324654 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:16.325080 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:16.325477 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:16.325482 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:16.325486 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:16.325624 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:16.326195 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:16.326200 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:16.326394 (+ 194us) client_negotiation.cc:769] Sending connection context
0812 00:59:16.326546 (+ 152us) client_negotiation.cc:240] Negotiation successful
0812 00:59:16.326935 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6102,"thread_start_us":5979,"threads_started":1}
I20260812 00:59:16.334646 2988 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:34245
I20260812 00:59:16.334668 3209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:34245 every 8 connection(s)
I20260812 00:59:16.336122 2988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 00:59:16.336638 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 2988
I20260812 00:59:16.336895 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance
I20260812 00:59:16.337781 22309 external_mini_cluster.cc:1428] Setting key 14e73edb9ac779ae0e5963184ce781a8
I20260812 00:59:16.338654 3212 heartbeater.cc:347] Connected to a master server at 127.21.201.126:42259
I20260812 00:59:16.338907 3212 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:16.339563 3212 heartbeater.cc:511] Master 127.21.201.126:42259 requested a full tablet report, sending...
I20260812 00:59:16.341969 2857 ts_manager.cc:194] Registered new tserver with Master: 8724c11a48bb4a7182aa973cd95bdb92 (127.21.201.65:34245)
I20260812 00:59:16.344151 2857 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.201.65@KRBTEST.COM'} at 127.21.201.65:34465
WARNING: no policy specified for oryx/127.21.201.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.201.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.201.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.66.keytab.
Entry for principal oryx/127.21.201.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.66.keytab.
I20260812 00:59:16.407938 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:0
--local_ip_for_outbound_sockets=127.21.201.66
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:42259
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.66.keytab
--principal=oryx/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:44081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:59:16.792346 3220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:16.792733 3220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:16.792898 3220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:16.801276 3220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:16.801455 3220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:16.801608 3220 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:59:16.801749 3220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 00:59:16.813859 3220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:44081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.66.keytab
--principal=oryx/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.201.126:42259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:16.815460 3220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:16.817008 3220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:16.831002 3226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:16.831108 3231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:16.831019 3225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:16.831688 3220 server_base.cc:1034] running on GCE node
I20260812 00:59:16.832405 3220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:16.833499 3220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:16.834756 3220 hybrid_clock.cc:648] HybridClock initialized: now 1786496356834691 us; error 41 us; skew 500 ppm
I20260812 00:59:16.835109 3220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:16 dist-test-slave-7f01 krb5kdc[2782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496356, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:16.838773 3220 init.cc:377] Logged in from keytab as oryx/127.21.201.66@KRBTEST.COM (short username oryx)
I20260812 00:59:16.840639 3220 webserver.cc:460] Webserver started at http://127.21.201.66:33851/ using document root <none> and password file <none>
I20260812 00:59:16.841425 3220 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:16.841521 3220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:16.841894 3220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:59:16.845120 3220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-1/data/instance:
uuid: "e0fa056511804a81a713d5e8a15d90c1"
format_stamp: "Formatted at 2026-08-12 00:59:16 on dist-test-slave-7f01"
server_key: "2739b345c39b0f31881a5d58d5f0d57a"
server_key_iv: "fddc350d602dd46af15c1ba182b75c2b"
server_key_version: "encryptionkey@0"
I20260812 00:59:16.846009 3220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance:
uuid: "e0fa056511804a81a713d5e8a15d90c1"
format_stamp: "Formatted at 2026-08-12 00:59:16 on dist-test-slave-7f01"
server_key: "2739b345c39b0f31881a5d58d5f0d57a"
server_key_iv: "fddc350d602dd46af15c1ba182b75c2b"
server_key_version: "encryptionkey@0"
I20260812 00:59:16.851945 3220 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 00:59:16.855841 3243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:16.857331 3220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 00:59:16.857532 3220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "e0fa056511804a81a713d5e8a15d90c1"
format_stamp: "Formatted at 2026-08-12 00:59:16 on dist-test-slave-7f01"
server_key: "2739b345c39b0f31881a5d58d5f0d57a"
server_key_iv: "fddc350d602dd46af15c1ba182b75c2b"
server_key_version: "encryptionkey@0"
I20260812 00:59:16.857769 3220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:16.876665 3220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:16.881222 3220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:16.881539 3220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:16.884573 3220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:59:16.884683 3220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:16.884819 3220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:59:16.884910 3220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 00:59:16 dist-test-slave-7f01 krb5kdc[2782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496356, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM
I20260812 00:59:16.915452 3260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:16.885844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:42259 (local address 127.21.201.66:59559)
0812 00:59:16.887178 (+ 1334us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:16.887265 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:16.889821 (+ 2556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:16.891030 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:16.891057 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:16.892080 (+ 1023us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:16.893151 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:16.893188 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:16.894878 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:16.894882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:16.896654 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:16.896668 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:16.897965 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:16.897981 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:16.898471 (+ 490us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:16.899511 (+ 1040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:16.899551 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:16.903905 (+ 4354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:16.908950 (+ 5045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:16.908964 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:16.909015 (+ 51us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:16.909816 (+ 801us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:16.910266 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:16.910271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:16.910275 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:16.910510 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:16.911722 (+ 1212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:16.911939 (+ 217us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:16.912535 (+ 596us) client_negotiation.cc:769] Sending connection context
0812 00:59:16.912996 (+ 461us) client_negotiation.cc:240] Negotiation successful
0812 00:59:16.913703 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"thread_start_us":511,"threads_started":1}
I20260812 00:59:16.916514 3259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:16.885198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:59559 (local address 127.21.201.126:42259)
0812 00:59:16.886141 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:16.886152 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:16.887452 (+ 1300us) server_negotiation.cc:407] Connection header received
0812 00:59:16.890235 (+ 2783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:16.890242 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:16.890485 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:16.890665 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:16.893477 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:16.894673 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:16.896818 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:16.897622 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:16.897819 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:16.904304 (+ 6485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:16.904315 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:16.904323 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:16.904386 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:16.908720 (+ 4334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:16.909978 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:16.909984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:16.909987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:16.910135 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:16.910644 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:16.910649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:16.910652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:16.911394 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:16.911737 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:16.915579 (+ 3842us) server_negotiation.cc:299] Negotiation successful
0812 00:59:16.916007 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":488,"threads_started":1}
I20260812 00:59:16.943429 3220 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:35809
I20260812 00:59:16.943470 3481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:35809 every 8 connection(s)
I20260812 00:59:16.944916 3220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 00:59:16.945699 3259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:16.926177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:36649 (local address 127.21.201.126:42259)
0812 00:59:16.926444 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:16.926454 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:16.927309 (+ 855us) server_negotiation.cc:407] Connection header received
0812 00:59:16.927598 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:16.927606 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:16.927720 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:16.927896 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:16.929003 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:16.930331 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:16.932157 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:16.932770 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:16.932947 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:16.937422 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:16.937432 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:16.937440 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:16.937490 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:16.941989 (+ 4499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:16.943250 (+ 1261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:16.943258 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:16.943263 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:16.943485 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:16.944007 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:16.944013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:16.944016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:16.944396 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:16.944596 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:16.945036 (+ 440us) server_negotiation.cc:299] Negotiation successful
0812 00:59:16.945320 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260812 00:59:16.945808 3414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:16.926223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:42259 (local address 127.21.201.66:36649)
0812 00:59:16.927170 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:16.927191 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:16.927428 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:16.927972 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:16.927980 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:16.928482 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:16.928818 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:16.928828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:16.930484 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:16.930489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:16.931989 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:16.932002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:16.933071 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:16.933082 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:16.933380 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:16.934687 (+ 1307us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:16.934732 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:16.936949 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:16.942215 (+ 5266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:16.942224 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:16.942232 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:16.943052 (+ 820us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:16.943664 (+ 612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:16.943669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:16.943672 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:16.943871 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:16.944536 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:16.944541 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:16.944756 (+ 215us) client_negotiation.cc:769] Sending connection context
0812 00:59:16.944916 (+ 160us) client_negotiation.cc:240] Negotiation successful
0812 00:59:16.945335 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"thread_start_us":597,"threads_started":1}
I20260812 00:59:16.947798 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 3220
I20260812 00:59:16.947991 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance
I20260812 00:59:16.948506 22309 external_mini_cluster.cc:1428] Setting key 0d13996fe9b1251ba2307772ffdaff50
I20260812 00:59:16.949146 3482 heartbeater.cc:347] Connected to a master server at 127.21.201.126:42259
I20260812 00:59:16.949435 3482 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:16.950363 3482 heartbeater.cc:511] Master 127.21.201.126:42259 requested a full tablet report, sending...
I20260812 00:59:16.952049 2857 ts_manager.cc:194] Registered new tserver with Master: e0fa056511804a81a713d5e8a15d90c1 (127.21.201.66:35809)
I20260812 00:59:16.952927 2857 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.201.66@KRBTEST.COM'} at 127.21.201.66:59559
WARNING: no policy specified for oryx/127.21.201.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.201.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.201.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.67.keytab.
Entry for principal oryx/127.21.201.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.67.keytab.
I20260812 00:59:17.017110 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:0
--local_ip_for_outbound_sockets=127.21.201.67
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:42259
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.67.keytab
--principal=oryx/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:44081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:59:17.347935 3212 heartbeater.cc:503] Master 127.21.201.126:42259 was elected leader, sending a full tablet report...
W20260812 00:59:17.387359 3486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:17.387683 3486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:17.387890 3486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:17.396200 3486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:17.396340 3486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:17.396502 3486 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:59:17.396638 3486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 00:59:17.407454 3486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:44081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.67.keytab
--principal=oryx/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.201.126:42259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:17.408946 3486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:17.410413 3486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:17.424854 3503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:17.424811 3501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:17.424906 3500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:17.425967 3486 server_base.cc:1034] running on GCE node
I20260812 00:59:17.426779 3486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:17.427970 3486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:17.429448 3486 hybrid_clock.cc:648] HybridClock initialized: now 1786496357429357 us; error 60 us; skew 500 ppm
I20260812 00:59:17.429751 3486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:17 dist-test-slave-7f01 krb5kdc[2782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496357, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:17.433275 3486 init.cc:377] Logged in from keytab as oryx/127.21.201.67@KRBTEST.COM (short username oryx)
I20260812 00:59:17.435088 3486 webserver.cc:460] Webserver started at http://127.21.201.67:35265/ using document root <none> and password file <none>
I20260812 00:59:17.435997 3486 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:17.436149 3486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:17.436522 3486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:59:17.439968 3486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-2/data/instance:
uuid: "01c6113ab1ef4026bccecc0f7633460d"
format_stamp: "Formatted at 2026-08-12 00:59:17 on dist-test-slave-7f01"
server_key: "6dd0891cd22e2966565f30d7fefb700e"
server_key_iv: "574af2630869824d0f342f464c972d8f"
server_key_version: "encryptionkey@0"
I20260812 00:59:17.440886 3486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance:
uuid: "01c6113ab1ef4026bccecc0f7633460d"
format_stamp: "Formatted at 2026-08-12 00:59:17 on dist-test-slave-7f01"
server_key: "6dd0891cd22e2966565f30d7fefb700e"
server_key_iv: "574af2630869824d0f342f464c972d8f"
server_key_version: "encryptionkey@0"
I20260812 00:59:17.446595 3486 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 00:59:17.450695 3520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:17.452212 3486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 00:59:17.452433 3486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "01c6113ab1ef4026bccecc0f7633460d"
format_stamp: "Formatted at 2026-08-12 00:59:17 on dist-test-slave-7f01"
server_key: "6dd0891cd22e2966565f30d7fefb700e"
server_key_iv: "574af2630869824d0f342f464c972d8f"
server_key_version: "encryptionkey@0"
I20260812 00:59:17.452723 3486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:17.476406 3486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:17.482368 3486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:17.482803 3486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:17.485893 3486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:59:17.486032 3486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:17.486234 3486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:59:17.486331 3486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 00:59:17 dist-test-slave-7f01 krb5kdc[2782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496357, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM
I20260812 00:59:17.514588 3538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:17.486553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:56031 (local address 127.21.201.126:42259)
0812 00:59:17.489261 (+ 2708us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:17.489268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:17.489294 (+ 26us) server_negotiation.cc:407] Connection header received
0812 00:59:17.490932 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:17.490938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:17.491037 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:17.491168 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:17.493772 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.494961 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.497865 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.498597 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.498901 (+ 304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:17.505054 (+ 6153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:17.505063 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:17.505071 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:17.505132 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:17.509164 (+ 4032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:17.511102 (+ 1938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:17.511108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:17.511113 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:17.511324 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:17.512173 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:17.512181 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:17.512187 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:17.512545 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:17.512766 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:17.514038 (+ 1272us) server_negotiation.cc:299] Negotiation successful
0812 00:59:17.514261 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2569,"thread_start_us":2425,"threads_started":1}
I20260812 00:59:17.516852 3539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:17.487274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:42259 (local address 127.21.201.67:56031)
0812 00:59:17.488332 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:17.488400 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:17.490554 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:17.491439 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:17.491447 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:17.492383 (+ 936us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:17.493457 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:17.493496 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.495148 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.495157 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:17.497690 (+ 2533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:17.497708 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.499084 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.499095 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:17.499668 (+ 573us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:17.500822 (+ 1154us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:17.500866 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:17.504695 (+ 3829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:17.509554 (+ 4859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:17.509574 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:17.509602 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:17.510900 (+ 1298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:17.511498 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:17.511510 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:17.511520 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:17.511956 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:17.512765 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:17.512775 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:17.513461 (+ 686us) client_negotiation.cc:769] Sending connection context
0812 00:59:17.513923 (+ 462us) client_negotiation.cc:240] Negotiation successful
0812 00:59:17.514759 (+ 836us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":513,"threads_started":1}
I20260812 00:59:17.543466 3538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:17.523660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:35543 (local address 127.21.201.126:42259)
0812 00:59:17.523865 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:17.523869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:17.524852 (+ 983us) server_negotiation.cc:407] Connection header received
0812 00:59:17.524995 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:17.525022 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:17.525127 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:17.525262 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:17.527115 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.528450 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.529979 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.530809 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.530989 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:17.534425 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:17.534435 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:17.534445 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:17.534495 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:17.539370 (+ 4875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:17.540632 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:17.540640 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:17.540646 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:17.540848 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:17.541273 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:17.541281 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:17.541285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:17.541740 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:17.541945 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:17.542607 (+ 662us) server_negotiation.cc:299] Negotiation successful
0812 00:59:17.542921 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 00:59:17.543656 3631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:17.523771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:42259 (local address 127.21.201.67:35543)
0812 00:59:17.524717 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:17.524738 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:17.524878 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:17.525328 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:17.525340 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:17.526289 (+ 949us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:17.526863 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:17.526890 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.528648 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.528659 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:17.529768 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:17.529781 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.531169 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.531182 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:17.531438 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:17.532353 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:17.532389 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:17.534145 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:17.539601 (+ 5456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:17.539612 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:17.539617 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:17.540460 (+ 843us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:17.540985 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:17.540990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:17.540994 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:17.541133 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:17.541902 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:17.541910 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:17.542203 (+ 293us) client_negotiation.cc:769] Sending connection context
0812 00:59:17.542443 (+ 240us) client_negotiation.cc:240] Negotiation successful
0812 00:59:17.542947 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":626,"threads_started":1}
I20260812 00:59:17.548736 3486 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:33863
I20260812 00:59:17.548748 3744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:33863 every 8 connection(s)
I20260812 00:59:17.550098 3486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 00:59:17.550730 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 3486
I20260812 00:59:17.550923 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance
I20260812 00:59:17.551671 22309 external_mini_cluster.cc:1428] Setting key 47faa336f804034c7c751afdd4d15a24
I20260812 00:59:17.552637 3745 heartbeater.cc:347] Connected to a master server at 127.21.201.126:42259
I20260812 00:59:17.552927 3745 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:17.553772 3745 heartbeater.cc:511] Master 127.21.201.126:42259 requested a full tablet report, sending...
I20260812 00:59:17.555177 2857 ts_manager.cc:194] Registered new tserver with Master: 01c6113ab1ef4026bccecc0f7633460d (127.21.201.67:33863)
I20260812 00:59:17.556073 2857 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.201.67@KRBTEST.COM'} at 127.21.201.67:56031
I20260812 00:59:17.568085 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 00:59:17 dist-test-slave-7f01 krb5kdc[2782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496357, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 00:59:17 dist-test-slave-7f01 krb5kdc[2782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:17 dist-test-slave-7f01 krb5kdc[2782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 00:59:17.638460 3538 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:17.625792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44186 (local address 127.21.201.126:42259)
0812 00:59:17.626061 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:17.626083 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:17.626615 (+ 532us) server_negotiation.cc:407] Connection header received
0812 00:59:17.626830 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:17.626834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:17.626908 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:17.627020 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:17.628857 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.629643 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.630931 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.631659 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.631858 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:17.637996 (+ 6138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44186: BlockingRecv error: recv got EOF from 127.0.0.1:44186 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 12 00:59:17 dist-test-slave-7f01 krb5kdc[2782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496357, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM
I20260812 00:59:17.659690 3538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:17.643166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44196 (local address 127.21.201.126:42259)
0812 00:59:17.643367 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:17.643371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:17.643586 (+ 215us) server_negotiation.cc:407] Connection header received
0812 00:59:17.643782 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:17.643789 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:17.643866 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:17.643978 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:17.645378 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.646373 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.647878 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.648528 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.648645 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:17.653023 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:17.653052 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:17.653055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:17.653098 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:17.656014 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:17.657380 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:17.657387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:17.657390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:17.657671 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:17.658221 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:17.658225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:17.658228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:17.658564 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:17.658755 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:17.659225 (+ 470us) server_negotiation.cc:299] Negotiation successful
0812 00:59:17.659427 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 00:59:17.665403 2857 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:59:17.674507 2857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:59:17.709367 3774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:17.690190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:35809 (local address 127.0.0.1:39884)
0812 00:59:17.699478 (+ 9288us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:17.699504 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:17.699647 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:17.701286 (+ 1639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:17.701293 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:17.701377 (+ 84us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:17.702013 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:17.702025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.703980 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.703988 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:17.706578 (+ 2590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:17.706598 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.708573 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.708579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:17.708758 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:17.708787 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 00:59:17.708844 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 00:59:17.708970 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9091,"mutex_wait_us":125,"spinlock_wait_cycles":17536,"thread_start_us":6528,"threads_started":1}
I20260812 00:59:17.710136 3777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:17.690348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56896 (local address 127.21.201.65:34245)
0812 00:59:17.700365 (+ 10017us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:17.700387 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:17.700427 (+ 40us) server_negotiation.cc:407] Connection header received
0812 00:59:17.700555 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:17.700563 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:17.700840 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:17.701130 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:17.702187 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.703802 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.706285 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.707829 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.709383 (+ 1554us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:17.709651 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:17.709740 (+ 89us) server_negotiation.cc:299] Negotiation successful
0812 00:59:17.709904 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9851,"thread_start_us":9692,"threads_started":1}
I20260812 00:59:17.710451 3775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:17.690127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:34245 (local address 127.0.0.1:56896)
0812 00:59:17.697150 (+ 7023us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:17.697192 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:17.697404 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:17.701287 (+ 3883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:17.701296 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:17.701393 (+ 97us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:17.701942 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:17.701962 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.703951 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.703957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:17.706089 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:17.706102 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.708174 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.708181 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:17.708415 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:17.708501 (+ 86us) client_negotiation.cc:769] Sending connection context
0812 00:59:17.708621 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 00:59:17.708829 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6750,"mutex_wait_us":118,"spinlock_wait_cycles":102144,"thread_start_us":9424,"threads_started":1}
I20260812 00:59:17.710897 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:17.690712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39884 (local address 127.21.201.66:35809)
0812 00:59:17.700667 (+ 9955us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:17.700675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:17.700700 (+ 25us) server_negotiation.cc:407] Connection header received
0812 00:59:17.700802 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:17.700809 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:17.701024 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:17.701765 (+ 741us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:17.702206 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.703819 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.706795 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.708257 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.710088 (+ 1831us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:17.710292 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:17.710371 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 00:59:17.710561 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9806,"thread_start_us":9667,"threads_started":1}
I20260812 00:59:17.710973 3781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:17.697618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:33863 (local address 127.0.0.1:59738)
0812 00:59:17.701094 (+ 3476us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:17.701114 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:17.701365 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:17.702029 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:17.702037 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:17.702068 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:17.702453 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:17.702465 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.704599 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.704603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:17.706912 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:17.706923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.710115 (+ 3192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.710122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:17.710296 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:17.710331 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 00:59:17.710465 (+ 134us) client_negotiation.cc:240] Negotiation successful
0812 00:59:17.710599 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3364,"mutex_wait_us":3,"spinlock_wait_cycles":9344,"thread_start_us":1630,"threads_started":1}
I20260812 00:59:17.711863 3780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:17.698151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59738 (local address 127.21.201.67:33863)
0812 00:59:17.699687 (+ 1536us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:17.699699 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:17.701385 (+ 1686us) server_negotiation.cc:407] Connection header received
0812 00:59:17.701531 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:17.701538 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:17.701776 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:17.701961 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:17.702701 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.704453 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.707074 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.708918 (+ 1844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.709128 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:17.709447 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:17.711227 (+ 1780us) server_negotiation.cc:299] Negotiation successful
0812 00:59:17.711449 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1357,"thread_start_us":1177,"threads_started":1}
I20260812 00:59:17.714654 3091 tablet_service.cc:1467] Processing CreateTablet for tablet 524e175062744ce4a3124e1bc5a9cc4a (DEFAULT_TABLE table=test-table [id=d1cabd8b662f48a08b667ffdea47d983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:59:17.714648 3336 tablet_service.cc:1467] Processing CreateTablet for tablet 524e175062744ce4a3124e1bc5a9cc4a (DEFAULT_TABLE table=test-table [id=d1cabd8b662f48a08b667ffdea47d983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:59:17.714941 3607 tablet_service.cc:1467] Processing CreateTablet for tablet 524e175062744ce4a3124e1bc5a9cc4a (DEFAULT_TABLE table=test-table [id=d1cabd8b662f48a08b667ffdea47d983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:59:17.716238 3091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 524e175062744ce4a3124e1bc5a9cc4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:17.716248 3336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 524e175062744ce4a3124e1bc5a9cc4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:17.716360 3607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 524e175062744ce4a3124e1bc5a9cc4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:17.732295 3782 tablet_bootstrap.cc:492] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d: Bootstrap starting.
I20260812 00:59:17.737221 3782 tablet_bootstrap.cc:654] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:17.738337 3785 tablet_bootstrap.cc:492] T 524e175062744ce4a3124e1bc5a9cc4a P 8724c11a48bb4a7182aa973cd95bdb92: Bootstrap starting.
I20260812 00:59:17.738540 3782 log.cc:824] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d: Log is configured to *not* fsync() on all Append() calls
I20260812 00:59:17.741297 3784 tablet_bootstrap.cc:492] T 524e175062744ce4a3124e1bc5a9cc4a P e0fa056511804a81a713d5e8a15d90c1: Bootstrap starting.
I20260812 00:59:17.742753 3785 tablet_bootstrap.cc:654] T 524e175062744ce4a3124e1bc5a9cc4a P 8724c11a48bb4a7182aa973cd95bdb92: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:17.743943 3782 tablet_bootstrap.cc:492] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d: No bootstrap required, opened a new log
I20260812 00:59:17.744313 3785 log.cc:824] T 524e175062744ce4a3124e1bc5a9cc4a P 8724c11a48bb4a7182aa973cd95bdb92: Log is configured to *not* fsync() on all Append() calls
I20260812 00:59:17.744793 3782 ts_tablet_manager.cc:1397] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260812 00:59:17.745726 3784 tablet_bootstrap.cc:654] T 524e175062744ce4a3124e1bc5a9cc4a P e0fa056511804a81a713d5e8a15d90c1: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:17.747059 3784 log.cc:824] T 524e175062744ce4a3124e1bc5a9cc4a P e0fa056511804a81a713d5e8a15d90c1: Log is configured to *not* fsync() on all Append() calls
I20260812 00:59:17.747202 3785 tablet_bootstrap.cc:492] T 524e175062744ce4a3124e1bc5a9cc4a P 8724c11a48bb4a7182aa973cd95bdb92: No bootstrap required, opened a new log
I20260812 00:59:17.747486 3785 ts_tablet_manager.cc:1397] T 524e175062744ce4a3124e1bc5a9cc4a P 8724c11a48bb4a7182aa973cd95bdb92: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260812 00:59:17.750432 3784 tablet_bootstrap.cc:492] T 524e175062744ce4a3124e1bc5a9cc4a P e0fa056511804a81a713d5e8a15d90c1: No bootstrap required, opened a new log
I20260812 00:59:17.750734 3784 ts_tablet_manager.cc:1397] T 524e175062744ce4a3124e1bc5a9cc4a P e0fa056511804a81a713d5e8a15d90c1: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260812 00:59:17.755899 3785 raft_consensus.cc:348] T 524e175062744ce4a3124e1bc5a9cc4a P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:17.756441 3785 raft_consensus.cc:374] T 524e175062744ce4a3124e1bc5a9cc4a P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:17.756182 3782 raft_consensus.cc:348] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:17.756582 3785 raft_consensus.cc:729] T 524e175062744ce4a3124e1bc5a9cc4a P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8724c11a48bb4a7182aa973cd95bdb92, State: Initialized, Role: FOLLOWER
I20260812 00:59:17.756814 3782 raft_consensus.cc:374] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:17.756985 3782 raft_consensus.cc:729] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c6113ab1ef4026bccecc0f7633460d, State: Initialized, Role: FOLLOWER
I20260812 00:59:17.757491 3785 consensus_queue.cc:260] T 524e175062744ce4a3124e1bc5a9cc4a P 8724c11a48bb4a7182aa973cd95bdb92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:17.758210 3782 consensus_queue.cc:260] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:17.759565 3745 heartbeater.cc:503] Master 127.21.201.126:42259 was elected leader, sending a full tablet report...
I20260812 00:59:17.760249 3782 ts_tablet_manager.cc:1428] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260812 00:59:17.761919 3785 ts_tablet_manager.cc:1428] T 524e175062744ce4a3124e1bc5a9cc4a P 8724c11a48bb4a7182aa973cd95bdb92: Time spent starting tablet: real 0.014s user 0.003s sys 0.011s
I20260812 00:59:17.763792 3784 raft_consensus.cc:348] T 524e175062744ce4a3124e1bc5a9cc4a P e0fa056511804a81a713d5e8a15d90c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:17.764382 3784 raft_consensus.cc:374] T 524e175062744ce4a3124e1bc5a9cc4a P e0fa056511804a81a713d5e8a15d90c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:17.764509 3784 raft_consensus.cc:729] T 524e175062744ce4a3124e1bc5a9cc4a P e0fa056511804a81a713d5e8a15d90c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0fa056511804a81a713d5e8a15d90c1, State: Initialized, Role: FOLLOWER
I20260812 00:59:17.765177 3784 consensus_queue.cc:260] T 524e175062744ce4a3124e1bc5a9cc4a P e0fa056511804a81a713d5e8a15d90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:17.767644 3482 heartbeater.cc:503] Master 127.21.201.126:42259 was elected leader, sending a full tablet report...
I20260812 00:59:17.768229 3784 ts_tablet_manager.cc:1428] T 524e175062744ce4a3124e1bc5a9cc4a P e0fa056511804a81a713d5e8a15d90c1: Time spent starting tablet: real 0.017s user 0.010s sys 0.006s
W20260812 00:59:17.802525 3746 tablet.cc:2369] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:59:17.833192 3795 raft_consensus.cc:482] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:59:17.833604 3795 raft_consensus.cc:504] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:17.835614 3795 leader_election.cc:288] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8724c11a48bb4a7182aa973cd95bdb92 (127.21.201.65:34245), e0fa056511804a81a713d5e8a15d90c1 (127.21.201.66:35809)
W20260812 00:59:17.839478 3215 tablet.cc:2369] T 524e175062744ce4a3124e1bc5a9cc4a P 8724c11a48bb4a7182aa973cd95bdb92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:59:17.842908 3777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:17.836014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:54303 (local address 127.21.201.65:34245)
0812 00:59:17.836290 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:17.836296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:17.836321 (+ 25us) server_negotiation.cc:407] Connection header received
0812 00:59:17.836514 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:17.836523 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:17.836627 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:17.836781 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:17.837297 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.838317 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.840690 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.841741 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.841929 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:17.842025 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:17.842589 (+ 564us) server_negotiation.cc:299] Negotiation successful
0812 00:59:17.842704 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260812 00:59:17.843151 3539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:17.835894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:34245 (local address 127.21.201.67:54303)
0812 00:59:17.836131 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:17.836158 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:17.836311 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:17.836818 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:17.836824 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:17.836863 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:17.837106 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:17.837120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.838511 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.838515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:17.840512 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:17.840526 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.842046 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.842055 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:17.842313 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:17.842364 (+ 51us) client_negotiation.cc:769] Sending connection context
0812 00:59:17.842541 (+ 177us) client_negotiation.cc:240] Negotiation successful
0812 00:59:17.842763 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":48}
I20260812 00:59:17.844456 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:17.836389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:57435 (local address 127.21.201.66:35809)
0812 00:59:17.836600 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:17.836609 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:17.837263 (+ 654us) server_negotiation.cc:407] Connection header received
0812 00:59:17.837348 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:17.837355 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:17.837438 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:17.837562 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:17.838031 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.838979 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.841496 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.842978 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.843248 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:17.843350 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:17.844057 (+ 707us) server_negotiation.cc:299] Negotiation successful
0812 00:59:17.844240 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 00:59:17.844698 3797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:17.836269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:35809 (local address 127.21.201.67:57435)
0812 00:59:17.837106 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:17.837126 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:17.837251 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:17.837577 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:17.837580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:17.837598 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:17.837881 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:17.837888 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.839153 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.839157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:17.841106 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:17.841123 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.843480 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.843489 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:17.843699 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:17.843865 (+ 166us) client_negotiation.cc:769] Sending connection context
0812 00:59:17.843981 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 00:59:17.844231 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":3,"thread_start_us":467,"threads_started":1}
I20260812 00:59:17.844815 3135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "524e175062744ce4a3124e1bc5a9cc4a" candidate_uuid: "01c6113ab1ef4026bccecc0f7633460d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8724c11a48bb4a7182aa973cd95bdb92" is_pre_election: true
I20260812 00:59:17.845844 3135 raft_consensus.cc:2454] T 524e175062744ce4a3124e1bc5a9cc4a P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01c6113ab1ef4026bccecc0f7633460d in term 0.
I20260812 00:59:17.845927 3385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "524e175062744ce4a3124e1bc5a9cc4a" candidate_uuid: "01c6113ab1ef4026bccecc0f7633460d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0fa056511804a81a713d5e8a15d90c1" is_pre_election: true
I20260812 00:59:17.846748 3523 leader_election.cc:302] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01c6113ab1ef4026bccecc0f7633460d, 8724c11a48bb4a7182aa973cd95bdb92; no voters:
I20260812 00:59:17.846839 3385 raft_consensus.cc:2454] T 524e175062744ce4a3124e1bc5a9cc4a P e0fa056511804a81a713d5e8a15d90c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01c6113ab1ef4026bccecc0f7633460d in term 0.
I20260812 00:59:17.847316 3795 raft_consensus.cc:2781] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:59:17.847513 3795 raft_consensus.cc:482] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:59:17.847824 3795 raft_consensus.cc:3037] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:17.851277 3795 raft_consensus.cc:504] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:17.852259 3795 leader_election.cc:288] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d [CANDIDATE]: Term 1 election: Requested vote from peers 8724c11a48bb4a7182aa973cd95bdb92 (127.21.201.65:34245), e0fa056511804a81a713d5e8a15d90c1 (127.21.201.66:35809)
I20260812 00:59:17.852807 3135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "524e175062744ce4a3124e1bc5a9cc4a" candidate_uuid: "01c6113ab1ef4026bccecc0f7633460d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8724c11a48bb4a7182aa973cd95bdb92"
I20260812 00:59:17.852892 3385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "524e175062744ce4a3124e1bc5a9cc4a" candidate_uuid: "01c6113ab1ef4026bccecc0f7633460d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0fa056511804a81a713d5e8a15d90c1"
I20260812 00:59:17.853082 3135 raft_consensus.cc:3037] T 524e175062744ce4a3124e1bc5a9cc4a P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:17.853178 3385 raft_consensus.cc:3037] T 524e175062744ce4a3124e1bc5a9cc4a P e0fa056511804a81a713d5e8a15d90c1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:17.855634 3135 raft_consensus.cc:2454] T 524e175062744ce4a3124e1bc5a9cc4a P 8724c11a48bb4a7182aa973cd95bdb92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01c6113ab1ef4026bccecc0f7633460d in term 1.
I20260812 00:59:17.855649 3385 raft_consensus.cc:2454] T 524e175062744ce4a3124e1bc5a9cc4a P e0fa056511804a81a713d5e8a15d90c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01c6113ab1ef4026bccecc0f7633460d in term 1.
I20260812 00:59:17.856249 3527 leader_election.cc:302] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01c6113ab1ef4026bccecc0f7633460d, e0fa056511804a81a713d5e8a15d90c1; no voters:
I20260812 00:59:17.856630 3795 raft_consensus.cc:2781] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:59:17.857162 3795 raft_consensus.cc:686] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d [term 1 LEADER]: Becoming Leader. State: Replica: 01c6113ab1ef4026bccecc0f7633460d, State: Running, Role: LEADER
I20260812 00:59:17.857780 3795 consensus_queue.cc:237] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:17.863008 2857 catalog_manager.cc:5495] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d reported cstate change: term changed from 0 to 1, leader changed from <none> to 01c6113ab1ef4026bccecc0f7633460d (127.21.201.67). New cstate: current_term: 1 leader_uuid: "01c6113ab1ef4026bccecc0f7633460d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } health_report { overall_health: HEALTHY } } }
I20260812 00:59:17.902226 3780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:17.895492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59752 (local address 127.21.201.67:33863)
0812 00:59:17.895715 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:17.895719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:17.895915 (+ 196us) server_negotiation.cc:407] Connection header received
0812 00:59:17.896183 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:17.896191 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:17.896344 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:17.896563 (+ 219us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:59:17.897029 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.897827 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.899491 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:17.900003 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:17.900114 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:17.900611 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:59:17.901216 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:59:17.901339 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:17.901609 (+ 270us) server_negotiation.cc:299] Negotiation successful
0812 00:59:17.901804 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 00:59:17.911242 3385 raft_consensus.cc:1261] T 524e175062744ce4a3124e1bc5a9cc4a P e0fa056511804a81a713d5e8a15d90c1 [term 1 FOLLOWER]: Refusing update from remote peer 01c6113ab1ef4026bccecc0f7633460d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:59:17.911244 3135 raft_consensus.cc:1261] T 524e175062744ce4a3124e1bc5a9cc4a P 8724c11a48bb4a7182aa973cd95bdb92 [term 1 FOLLOWER]: Refusing update from remote peer 01c6113ab1ef4026bccecc0f7633460d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:59:17.912508 3798 consensus_queue.cc:1035] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:59:17.913174 3795 consensus_queue.cc:1035] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:59:17.943997 2856 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:59:17.944561 2856 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:59:17.949769 2856 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e: Sending DeleteTablet for 3 replicas of tablet 524e175062744ce4a3124e1bc5a9cc4a
I20260812 00:59:17.951392 3336 tablet_service.cc:1514] Processing DeleteTablet for tablet 524e175062744ce4a3124e1bc5a9cc4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:59:17 UTC) from {username='oryx', principal='oryx/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:39884
I20260812 00:59:17.951385 3091 tablet_service.cc:1514] Processing DeleteTablet for tablet 524e175062744ce4a3124e1bc5a9cc4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:59:17 UTC) from {username='oryx', principal='oryx/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:56896
I20260812 00:59:17.952678 3607 tablet_service.cc:1514] Processing DeleteTablet for tablet 524e175062744ce4a3124e1bc5a9cc4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:59:17 UTC) from {username='oryx', principal='oryx/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:59738
I20260812 00:59:17.962880 3827 tablet_replica.cc:330] T 524e175062744ce4a3124e1bc5a9cc4a P 8724c11a48bb4a7182aa973cd95bdb92: stopping tablet replica
I20260812 00:59:17.963032 2856 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:59:17.963982 2856 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:59:17.964035 3827 raft_consensus.cc:2229] T 524e175062744ce4a3124e1bc5a9cc4a P 8724c11a48bb4a7182aa973cd95bdb92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:59:17.965606 3827 raft_consensus.cc:2258] T 524e175062744ce4a3124e1bc5a9cc4a P 8724c11a48bb4a7182aa973cd95bdb92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:59:17.965802 3829 tablet_replica.cc:330] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d: stopping tablet replica
I20260812 00:59:17.966292 3829 raft_consensus.cc:2229] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:59:17.967235 3829 raft_consensus.cc:2258] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:59:17.968065 3825 tablet_replica.cc:330] T 524e175062744ce4a3124e1bc5a9cc4a P e0fa056511804a81a713d5e8a15d90c1: stopping tablet replica
I20260812 00:59:17.968502 3825 raft_consensus.cc:2229] T 524e175062744ce4a3124e1bc5a9cc4a P e0fa056511804a81a713d5e8a15d90c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:59:17.969244 3825 raft_consensus.cc:2258] T 524e175062744ce4a3124e1bc5a9cc4a P e0fa056511804a81a713d5e8a15d90c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:59:17.972079 3825 ts_tablet_manager.cc:1905] T 524e175062744ce4a3124e1bc5a9cc4a P e0fa056511804a81a713d5e8a15d90c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:59:17.972517 3827 ts_tablet_manager.cc:1905] T 524e175062744ce4a3124e1bc5a9cc4a P 8724c11a48bb4a7182aa973cd95bdb92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:59:17.974957 3091 tablet_service.cc:1467] Processing CreateTablet for tablet ffacabaabc274141999ff8d008395041 (DEFAULT_TABLE table=test-table [id=bdab375b68c74eb5b36c5aff39d831c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:59:17.974969 3607 tablet_service.cc:1467] Processing CreateTablet for tablet ffacabaabc274141999ff8d008395041 (DEFAULT_TABLE table=test-table [id=bdab375b68c74eb5b36c5aff39d831c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:59:17.975453 3091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffacabaabc274141999ff8d008395041. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:17.975716 3607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffacabaabc274141999ff8d008395041. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:17.979435 3825 ts_tablet_manager.cc:1918] T 524e175062744ce4a3124e1bc5a9cc4a P e0fa056511804a81a713d5e8a15d90c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:59:17.979700 3825 log.cc:1196] T 524e175062744ce4a3124e1bc5a9cc4a P e0fa056511804a81a713d5e8a15d90c1: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/524e175062744ce4a3124e1bc5a9cc4a
I20260812 00:59:17.980177 3336 tablet_service.cc:1467] Processing CreateTablet for tablet ffacabaabc274141999ff8d008395041 (DEFAULT_TABLE table=test-table [id=bdab375b68c74eb5b36c5aff39d831c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:59:17.980566 3825 ts_tablet_manager.cc:1939] T 524e175062744ce4a3124e1bc5a9cc4a P e0fa056511804a81a713d5e8a15d90c1: Deleting consensus metadata
I20260812 00:59:17.980672 3336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffacabaabc274141999ff8d008395041. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:17.982748 2840 catalog_manager.cc:4841] TS e0fa056511804a81a713d5e8a15d90c1 (127.21.201.66:35809): tablet 524e175062744ce4a3124e1bc5a9cc4a (table test-table [id=d1cabd8b662f48a08b667ffdea47d983]) successfully deleted
I20260812 00:59:17.983032 3782 tablet_bootstrap.cc:492] T ffacabaabc274141999ff8d008395041 P 01c6113ab1ef4026bccecc0f7633460d: Bootstrap starting.
I20260812 00:59:17.984818 3782 tablet_bootstrap.cc:654] T ffacabaabc274141999ff8d008395041 P 01c6113ab1ef4026bccecc0f7633460d: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:17.985937 3785 tablet_bootstrap.cc:492] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92: Bootstrap starting.
I20260812 00:59:17.987587 3784 tablet_bootstrap.cc:492] T ffacabaabc274141999ff8d008395041 P e0fa056511804a81a713d5e8a15d90c1: Bootstrap starting.
I20260812 00:59:17.988337 3785 tablet_bootstrap.cc:654] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:17.989240 3784 tablet_bootstrap.cc:654] T ffacabaabc274141999ff8d008395041 P e0fa056511804a81a713d5e8a15d90c1: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:17.991343 3829 ts_tablet_manager.cc:1905] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:59:17.994601 3785 tablet_bootstrap.cc:492] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92: No bootstrap required, opened a new log
I20260812 00:59:17.994830 3785 ts_tablet_manager.cc:1397] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20260812 00:59:17.995846 3785 raft_consensus.cc:348] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:17.996232 3785 raft_consensus.cc:374] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:17.996398 3785 raft_consensus.cc:729] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8724c11a48bb4a7182aa973cd95bdb92, State: Initialized, Role: FOLLOWER
I20260812 00:59:17.996865 3785 consensus_queue.cc:260] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:17.997593 3785 ts_tablet_manager.cc:1428] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:59:18.000406 3782 tablet_bootstrap.cc:492] T ffacabaabc274141999ff8d008395041 P 01c6113ab1ef4026bccecc0f7633460d: No bootstrap required, opened a new log
I20260812 00:59:18.000649 3782 ts_tablet_manager.cc:1397] T ffacabaabc274141999ff8d008395041 P 01c6113ab1ef4026bccecc0f7633460d: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.007s
I20260812 00:59:18.001696 3782 raft_consensus.cc:348] T ffacabaabc274141999ff8d008395041 P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:18.002036 3782 raft_consensus.cc:374] T ffacabaabc274141999ff8d008395041 P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:18.002208 3782 raft_consensus.cc:729] T ffacabaabc274141999ff8d008395041 P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c6113ab1ef4026bccecc0f7633460d, State: Initialized, Role: FOLLOWER
I20260812 00:59:18.002612 3782 consensus_queue.cc:260] T ffacabaabc274141999ff8d008395041 P 01c6113ab1ef4026bccecc0f7633460d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:18.003165 3829 ts_tablet_manager.cc:1918] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:59:18.003473 3829 log.cc:1196] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/524e175062744ce4a3124e1bc5a9cc4a
I20260812 00:59:18.003912 3827 ts_tablet_manager.cc:1918] T 524e175062744ce4a3124e1bc5a9cc4a P 8724c11a48bb4a7182aa973cd95bdb92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:59:18.004317 3829 ts_tablet_manager.cc:1939] T 524e175062744ce4a3124e1bc5a9cc4a P 01c6113ab1ef4026bccecc0f7633460d: Deleting consensus metadata
I20260812 00:59:18.004550 3827 log.cc:1196] T 524e175062744ce4a3124e1bc5a9cc4a P 8724c11a48bb4a7182aa973cd95bdb92: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-0/wal/wals/524e175062744ce4a3124e1bc5a9cc4a
I20260812 00:59:18.005589 3784 tablet_bootstrap.cc:492] T ffacabaabc274141999ff8d008395041 P e0fa056511804a81a713d5e8a15d90c1: No bootstrap required, opened a new log
I20260812 00:59:18.005712 3782 ts_tablet_manager.cc:1428] T ffacabaabc274141999ff8d008395041 P 01c6113ab1ef4026bccecc0f7633460d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260812 00:59:18.005853 3784 ts_tablet_manager.cc:1397] T ffacabaabc274141999ff8d008395041 P e0fa056511804a81a713d5e8a15d90c1: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.004s
I20260812 00:59:18.006486 2840 catalog_manager.cc:4841] TS 01c6113ab1ef4026bccecc0f7633460d (127.21.201.67:33863): tablet 524e175062744ce4a3124e1bc5a9cc4a (table test-table [id=d1cabd8b662f48a08b667ffdea47d983]) successfully deleted
I20260812 00:59:18.007078 3827 ts_tablet_manager.cc:1939] T 524e175062744ce4a3124e1bc5a9cc4a P 8724c11a48bb4a7182aa973cd95bdb92: Deleting consensus metadata
I20260812 00:59:18.007449 3784 raft_consensus.cc:348] T ffacabaabc274141999ff8d008395041 P e0fa056511804a81a713d5e8a15d90c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:18.007795 3784 raft_consensus.cc:374] T ffacabaabc274141999ff8d008395041 P e0fa056511804a81a713d5e8a15d90c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:18.007952 3784 raft_consensus.cc:729] T ffacabaabc274141999ff8d008395041 P e0fa056511804a81a713d5e8a15d90c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0fa056511804a81a713d5e8a15d90c1, State: Initialized, Role: FOLLOWER
I20260812 00:59:18.008294 3784 consensus_queue.cc:260] T ffacabaabc274141999ff8d008395041 P e0fa056511804a81a713d5e8a15d90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:18.009212 2835 catalog_manager.cc:4841] TS 8724c11a48bb4a7182aa973cd95bdb92 (127.21.201.65:34245): tablet 524e175062744ce4a3124e1bc5a9cc4a (table test-table [id=d1cabd8b662f48a08b667ffdea47d983]) successfully deleted
I20260812 00:59:18.009336 3784 ts_tablet_manager.cc:1428] T ffacabaabc274141999ff8d008395041 P e0fa056511804a81a713d5e8a15d90c1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260812 00:59:18.199807 3483 tablet.cc:2369] T ffacabaabc274141999ff8d008395041 P e0fa056511804a81a713d5e8a15d90c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:59:18.322093 3794 raft_consensus.cc:482] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:59:18.322419 3794 raft_consensus.cc:504] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:18.324647 3794 leader_election.cc:288] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0fa056511804a81a713d5e8a15d90c1 (127.21.201.66:35809), 01c6113ab1ef4026bccecc0f7633460d (127.21.201.67:33863)
I20260812 00:59:18.333307 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:18.325140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:47181 (local address 127.21.201.66:35809)
0812 00:59:18.325392 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:18.325400 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:18.325973 (+ 573us) server_negotiation.cc:407] Connection header received
0812 00:59:18.326339 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:18.326348 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:18.326457 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:18.326541 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:18.327497 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.328926 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.331028 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.332125 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.332316 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:18.332404 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:18.332814 (+ 410us) server_negotiation.cc:299] Negotiation successful
0812 00:59:18.332965 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 00:59:18.333668 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:18.325141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:35809 (local address 127.21.201.65:47181)
0812 00:59:18.325866 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:18.325889 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:18.326098 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:18.326843 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:18.326847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:18.326870 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:18.327284 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:18.327298 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.329121 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.329130 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:18.330831 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:18.330845 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.332388 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.332396 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:18.332600 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:18.332642 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 00:59:18.332758 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 00:59:18.332920 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":76,"thread_start_us":388,"threads_started":1}
I20260812 00:59:18.334205 3780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:18.326181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:40071 (local address 127.21.201.67:33863)
0812 00:59:18.326394 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:18.326401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:18.327110 (+ 709us) server_negotiation.cc:407] Connection header received
0812 00:59:18.327306 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:18.327312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:18.327415 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:18.327503 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:18.328115 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.328956 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.331631 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.332912 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.333073 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:18.333145 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:18.333754 (+ 609us) server_negotiation.cc:299] Negotiation successful
0812 00:59:18.333898 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 00:59:18.334409 3836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:18.326008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:33863 (local address 127.21.201.65:40071)
0812 00:59:18.326948 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:18.326973 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:18.327134 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:18.327515 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:18.327519 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:18.327544 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:18.327952 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:18.327962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.329150 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.329156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:18.331420 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:18.331437 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.333339 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.333348 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:18.333554 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:18.333588 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 00:59:18.333714 (+ 126us) client_negotiation.cc:240] Negotiation successful
0812 00:59:18.333866 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"mutex_wait_us":1,"thread_start_us":577,"threads_started":1}
I20260812 00:59:18.335088 3385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffacabaabc274141999ff8d008395041" candidate_uuid: "8724c11a48bb4a7182aa973cd95bdb92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0fa056511804a81a713d5e8a15d90c1" is_pre_election: true
I20260812 00:59:18.335403 3385 raft_consensus.cc:2454] T ffacabaabc274141999ff8d008395041 P e0fa056511804a81a713d5e8a15d90c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8724c11a48bb4a7182aa973cd95bdb92 in term 0.
I20260812 00:59:18.335341 3648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffacabaabc274141999ff8d008395041" candidate_uuid: "8724c11a48bb4a7182aa973cd95bdb92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01c6113ab1ef4026bccecc0f7633460d" is_pre_election: true
I20260812 00:59:18.335973 3648 raft_consensus.cc:2454] T ffacabaabc274141999ff8d008395041 P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8724c11a48bb4a7182aa973cd95bdb92 in term 0.
I20260812 00:59:18.336165 3017 leader_election.cc:302] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8724c11a48bb4a7182aa973cd95bdb92, e0fa056511804a81a713d5e8a15d90c1; no voters:
I20260812 00:59:18.336658 3794 raft_consensus.cc:2781] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:59:18.336807 3794 raft_consensus.cc:482] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:59:18.336908 3794 raft_consensus.cc:3037] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:18.340085 3794 raft_consensus.cc:504] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:18.341125 3794 leader_election.cc:288] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92 [CANDIDATE]: Term 1 election: Requested vote from peers e0fa056511804a81a713d5e8a15d90c1 (127.21.201.66:35809), 01c6113ab1ef4026bccecc0f7633460d (127.21.201.67:33863)
I20260812 00:59:18.341363 3385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffacabaabc274141999ff8d008395041" candidate_uuid: "8724c11a48bb4a7182aa973cd95bdb92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0fa056511804a81a713d5e8a15d90c1"
I20260812 00:59:18.341617 3385 raft_consensus.cc:3037] T ffacabaabc274141999ff8d008395041 P e0fa056511804a81a713d5e8a15d90c1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:18.344703 3385 raft_consensus.cc:2454] T ffacabaabc274141999ff8d008395041 P e0fa056511804a81a713d5e8a15d90c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8724c11a48bb4a7182aa973cd95bdb92 in term 1.
I20260812 00:59:18.345074 3648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffacabaabc274141999ff8d008395041" candidate_uuid: "8724c11a48bb4a7182aa973cd95bdb92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01c6113ab1ef4026bccecc0f7633460d"
I20260812 00:59:18.345329 3648 raft_consensus.cc:3037] T ffacabaabc274141999ff8d008395041 P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:18.347647 3648 raft_consensus.cc:2454] T ffacabaabc274141999ff8d008395041 P 01c6113ab1ef4026bccecc0f7633460d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8724c11a48bb4a7182aa973cd95bdb92 in term 1.
I20260812 00:59:18.348294 3017 leader_election.cc:302] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8724c11a48bb4a7182aa973cd95bdb92, e0fa056511804a81a713d5e8a15d90c1; no voters:
I20260812 00:59:18.348950 3794 raft_consensus.cc:2781] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:59:18.350518 3794 raft_consensus.cc:686] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92 [term 1 LEADER]: Becoming Leader. State: Replica: 8724c11a48bb4a7182aa973cd95bdb92, State: Running, Role: LEADER
I20260812 00:59:18.350999 3794 consensus_queue.cc:237] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:18.354856 2856 catalog_manager.cc:5495] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8724c11a48bb4a7182aa973cd95bdb92 (127.21.201.65). New cstate: current_term: 1 leader_uuid: "8724c11a48bb4a7182aa973cd95bdb92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } health_report { overall_health: UNKNOWN } } }
Aug 12 00:59:18 dist-test-slave-7f01 krb5kdc[2782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496355, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.126@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM
I20260812 00:59:18.487540 3852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:18.471392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:42259 (local address 127.0.0.1:44200)
0812 00:59:18.472429 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:18.472447 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:18.472586 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:18.472909 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:18.472912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:18.473624 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:18.473840 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:18.473849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.475030 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.475035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:18.476070 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:18.476082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.477071 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.477077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:18.477227 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:18.478189 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:18.478217 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:18.480855 (+ 2638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:18.484549 (+ 3694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:18.484562 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:18.484568 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:18.485026 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:18.485445 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:18.485449 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:18.485451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:18.485582 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:18.486509 (+ 927us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:18.486516 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:18.486712 (+ 196us) client_negotiation.cc:769] Sending connection context
0812 00:59:18.486878 (+ 166us) client_negotiation.cc:240] Negotiation successful
0812 00:59:18.487139 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"thread_start_us":749,"threads_started":1}
I20260812 00:59:18.487644 3851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:18.471407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44200 (local address 127.21.201.126:42259)
0812 00:59:18.472362 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:18.472368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:18.472545 (+ 177us) server_negotiation.cc:407] Connection header received
0812 00:59:18.472694 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:18.472697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:18.472758 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:18.472853 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:18.474048 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.474847 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.476282 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.476792 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.476947 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:18.481151 (+ 4204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:18.481158 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:18.481163 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:18.481207 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:18.484379 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:18.485188 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:18.485193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:18.485195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:18.485337 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:18.485729 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:18.485805 (+ 76us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:18.485816 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:18.486313 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:18.486530 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:18.487106 (+ 576us) server_negotiation.cc:299] Negotiation successful
0812 00:59:18.487292 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":466,"threads_started":1}
I20260812 00:59:18.493615 2856 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:44200:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 00:59:18.494854 2856 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:59:18.503206 3091 tablet_service.cc:1467] Processing CreateTablet for tablet f436ee197fd64fd88ca3d717be0c82f2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dd55d19755bb4cdeb851e57b759867e1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 00:59:18.503330 3336 tablet_service.cc:1467] Processing CreateTablet for tablet f436ee197fd64fd88ca3d717be0c82f2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dd55d19755bb4cdeb851e57b759867e1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 00:59:18.503422 3607 tablet_service.cc:1467] Processing CreateTablet for tablet f436ee197fd64fd88ca3d717be0c82f2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dd55d19755bb4cdeb851e57b759867e1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 00:59:18.503923 3091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f436ee197fd64fd88ca3d717be0c82f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:18.504015 3607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f436ee197fd64fd88ca3d717be0c82f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:18.504007 3336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f436ee197fd64fd88ca3d717be0c82f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:18.511897 3853 tablet_bootstrap.cc:492] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d: Bootstrap starting.
I20260812 00:59:18.512732 3854 tablet_bootstrap.cc:492] T f436ee197fd64fd88ca3d717be0c82f2 P e0fa056511804a81a713d5e8a15d90c1: Bootstrap starting.
I20260812 00:59:18.513504 3857 tablet_bootstrap.cc:492] T f436ee197fd64fd88ca3d717be0c82f2 P 8724c11a48bb4a7182aa973cd95bdb92: Bootstrap starting.
I20260812 00:59:18.514663 3854 tablet_bootstrap.cc:654] T f436ee197fd64fd88ca3d717be0c82f2 P e0fa056511804a81a713d5e8a15d90c1: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:18.514796 3853 tablet_bootstrap.cc:654] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:18.515504 3857 tablet_bootstrap.cc:654] T f436ee197fd64fd88ca3d717be0c82f2 P 8724c11a48bb4a7182aa973cd95bdb92: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:18.517228 3853 tablet_bootstrap.cc:492] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d: No bootstrap required, opened a new log
I20260812 00:59:18.517385 3854 tablet_bootstrap.cc:492] T f436ee197fd64fd88ca3d717be0c82f2 P e0fa056511804a81a713d5e8a15d90c1: No bootstrap required, opened a new log
I20260812 00:59:18.517409 3853 ts_tablet_manager.cc:1397] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260812 00:59:18.517544 3854 ts_tablet_manager.cc:1397] T f436ee197fd64fd88ca3d717be0c82f2 P e0fa056511804a81a713d5e8a15d90c1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260812 00:59:18.517871 3857 tablet_bootstrap.cc:492] T f436ee197fd64fd88ca3d717be0c82f2 P 8724c11a48bb4a7182aa973cd95bdb92: No bootstrap required, opened a new log
I20260812 00:59:18.518033 3857 ts_tablet_manager.cc:1397] T f436ee197fd64fd88ca3d717be0c82f2 P 8724c11a48bb4a7182aa973cd95bdb92: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260812 00:59:18.518260 3853 raft_consensus.cc:348] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:18.518268 3854 raft_consensus.cc:348] T f436ee197fd64fd88ca3d717be0c82f2 P e0fa056511804a81a713d5e8a15d90c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:18.518524 3854 raft_consensus.cc:374] T f436ee197fd64fd88ca3d717be0c82f2 P e0fa056511804a81a713d5e8a15d90c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:18.518519 3853 raft_consensus.cc:374] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:18.518620 3854 raft_consensus.cc:729] T f436ee197fd64fd88ca3d717be0c82f2 P e0fa056511804a81a713d5e8a15d90c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0fa056511804a81a713d5e8a15d90c1, State: Initialized, Role: FOLLOWER
I20260812 00:59:18.518628 3853 raft_consensus.cc:729] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c6113ab1ef4026bccecc0f7633460d, State: Initialized, Role: FOLLOWER
I20260812 00:59:18.518842 3857 raft_consensus.cc:348] T f436ee197fd64fd88ca3d717be0c82f2 P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:18.518878 3853 consensus_queue.cc:260] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:18.518868 3854 consensus_queue.cc:260] T f436ee197fd64fd88ca3d717be0c82f2 P e0fa056511804a81a713d5e8a15d90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:18.519086 3857 raft_consensus.cc:374] T f436ee197fd64fd88ca3d717be0c82f2 P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:18.519160 3857 raft_consensus.cc:729] T f436ee197fd64fd88ca3d717be0c82f2 P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8724c11a48bb4a7182aa973cd95bdb92, State: Initialized, Role: FOLLOWER
I20260812 00:59:18.519461 3857 consensus_queue.cc:260] T f436ee197fd64fd88ca3d717be0c82f2 P 8724c11a48bb4a7182aa973cd95bdb92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:18.520026 3794 tablet_replica.cc:439] T f436ee197fd64fd88ca3d717be0c82f2 P 8724c11a48bb4a7182aa973cd95bdb92: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } } }
I20260812 00:59:18.520344 3794 tablet_replica.cc:442] T f436ee197fd64fd88ca3d717be0c82f2 P 8724c11a48bb4a7182aa973cd95bdb92: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:59:18.521508 3857 ts_tablet_manager.cc:1428] T f436ee197fd64fd88ca3d717be0c82f2 P 8724c11a48bb4a7182aa973cd95bdb92: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:59:18.520184 3853 ts_tablet_manager.cc:1428] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:59:18.520529 3864 tablet_replica.cc:439] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } } }
I20260812 00:59:18.524433 3863 tablet_replica.cc:439] T f436ee197fd64fd88ca3d717be0c82f2 P e0fa056511804a81a713d5e8a15d90c1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } } }
I20260812 00:59:18.525207 3863 tablet_replica.cc:442] T f436ee197fd64fd88ca3d717be0c82f2 P e0fa056511804a81a713d5e8a15d90c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:59:18.525202 3864 tablet_replica.cc:442] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:59:18.525314 3854 ts_tablet_manager.cc:1428] T f436ee197fd64fd88ca3d717be0c82f2 P e0fa056511804a81a713d5e8a15d90c1: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20260812 00:59:18.525405 3864 raft_consensus.cc:482] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:59:18.525557 3864 raft_consensus.cc:504] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:18.526244 3864 leader_election.cc:288] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8724c11a48bb4a7182aa973cd95bdb92 (127.21.201.65:34245), e0fa056511804a81a713d5e8a15d90c1 (127.21.201.66:35809)
I20260812 00:59:18.527451 3385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f436ee197fd64fd88ca3d717be0c82f2" candidate_uuid: "01c6113ab1ef4026bccecc0f7633460d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0fa056511804a81a713d5e8a15d90c1" is_pre_election: true
I20260812 00:59:18.527809 3385 raft_consensus.cc:2454] T f436ee197fd64fd88ca3d717be0c82f2 P e0fa056511804a81a713d5e8a15d90c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01c6113ab1ef4026bccecc0f7633460d in term 0.
I20260812 00:59:18.528359 3527 leader_election.cc:302] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01c6113ab1ef4026bccecc0f7633460d, e0fa056511804a81a713d5e8a15d90c1; no voters:
I20260812 00:59:18.529026 3864 raft_consensus.cc:2781] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:59:18.529171 3864 raft_consensus.cc:482] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:59:18.529311 3864 raft_consensus.cc:3037] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:18.531709 3864 raft_consensus.cc:504] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:18.532891 3135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f436ee197fd64fd88ca3d717be0c82f2" candidate_uuid: "01c6113ab1ef4026bccecc0f7633460d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8724c11a48bb4a7182aa973cd95bdb92" is_pre_election: true
I20260812 00:59:18.532950 3385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f436ee197fd64fd88ca3d717be0c82f2" candidate_uuid: "01c6113ab1ef4026bccecc0f7633460d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0fa056511804a81a713d5e8a15d90c1"
I20260812 00:59:18.533056 3133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f436ee197fd64fd88ca3d717be0c82f2" candidate_uuid: "01c6113ab1ef4026bccecc0f7633460d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8724c11a48bb4a7182aa973cd95bdb92"
I20260812 00:59:18.533211 3385 raft_consensus.cc:3037] T f436ee197fd64fd88ca3d717be0c82f2 P e0fa056511804a81a713d5e8a15d90c1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:18.533236 3135 raft_consensus.cc:2454] T f436ee197fd64fd88ca3d717be0c82f2 P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01c6113ab1ef4026bccecc0f7633460d in term 0.
I20260812 00:59:18.535373 3864 leader_election.cc:288] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d [CANDIDATE]: Term 1 election: Requested vote from peers 8724c11a48bb4a7182aa973cd95bdb92 (127.21.201.65:34245), e0fa056511804a81a713d5e8a15d90c1 (127.21.201.66:35809)
I20260812 00:59:18.536394 3385 raft_consensus.cc:2454] T f436ee197fd64fd88ca3d717be0c82f2 P e0fa056511804a81a713d5e8a15d90c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01c6113ab1ef4026bccecc0f7633460d in term 1.
I20260812 00:59:18.536947 3527 leader_election.cc:302] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01c6113ab1ef4026bccecc0f7633460d, e0fa056511804a81a713d5e8a15d90c1; no voters: 8724c11a48bb4a7182aa973cd95bdb92
I20260812 00:59:18.537304 3864 raft_consensus.cc:2781] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:59:18.537462 3864 raft_consensus.cc:686] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d [term 1 LEADER]: Becoming Leader. State: Replica: 01c6113ab1ef4026bccecc0f7633460d, State: Running, Role: LEADER
I20260812 00:59:18.537771 3864 consensus_queue.cc:237] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:18.539386 3867 tablet_replica.cc:439] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d: TxnStatusTablet state changed. Reason: New leader 01c6113ab1ef4026bccecc0f7633460d. Latest consensus state: current_term: 1 leader_uuid: "01c6113ab1ef4026bccecc0f7633460d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } } }
I20260812 00:59:18.539793 3867 tablet_replica.cc:442] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:59:18.540624 3869 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 00:59:18.540905 3869 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 00:59:18.541109 2857 catalog_manager.cc:5495] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d reported cstate change: term changed from 0 to 1, leader changed from <none> to 01c6113ab1ef4026bccecc0f7633460d (127.21.201.67). New cstate: current_term: 1 leader_uuid: "01c6113ab1ef4026bccecc0f7633460d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } health_report { overall_health: HEALTHY } } }
I20260812 00:59:18.573658 3780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:18.566499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59766 (local address 127.21.201.67:33863)
0812 00:59:18.566723 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:18.566729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:18.567095 (+ 366us) server_negotiation.cc:407] Connection header received
0812 00:59:18.567389 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:18.567394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:18.567478 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:18.567579 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:59:18.568066 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.569261 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.571346 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.572082 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.572208 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:18.572837 (+ 629us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:59:18.572972 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:59:18.573059 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:18.573243 (+ 184us) server_negotiation.cc:299] Negotiation successful
0812 00:59:18.573338 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 00:59:18.573755 3852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:18.566552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:33863 (local address 127.0.0.1:59766)
0812 00:59:18.566977 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:18.567000 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:18.567176 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:18.567631 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:18.567636 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:18.567650 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:59:18.567912 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:18.567925 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.569481 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.569492 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:18.571172 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:18.571191 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.572271 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.572278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:18.572531 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:18.572623 (+ 92us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:59:18.573089 (+ 466us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:59:18.573113 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 00:59:18.573208 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 00:59:18.573380 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221}
I20260812 00:59:18.578207 3133 raft_consensus.cc:3037] T f436ee197fd64fd88ca3d717be0c82f2 P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:18.578382 3385 raft_consensus.cc:1261] T f436ee197fd64fd88ca3d717be0c82f2 P e0fa056511804a81a713d5e8a15d90c1 [term 1 FOLLOWER]: Refusing update from remote peer 01c6113ab1ef4026bccecc0f7633460d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:59:18.579188 3867 consensus_queue.cc:1035] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:59:18.582212 3133 raft_consensus.cc:1261] T f436ee197fd64fd88ca3d717be0c82f2 P 8724c11a48bb4a7182aa973cd95bdb92 [term 1 FOLLOWER]: Refusing update from remote peer 01c6113ab1ef4026bccecc0f7633460d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:59:18.582954 3867 consensus_queue.cc:1035] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:59:18.586163 3863 tablet_replica.cc:439] T f436ee197fd64fd88ca3d717be0c82f2 P e0fa056511804a81a713d5e8a15d90c1: TxnStatusTablet state changed. Reason: New leader 01c6113ab1ef4026bccecc0f7633460d. Latest consensus state: current_term: 1 leader_uuid: "01c6113ab1ef4026bccecc0f7633460d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } } }
I20260812 00:59:18.586457 3863 tablet_replica.cc:442] T f436ee197fd64fd88ca3d717be0c82f2 P e0fa056511804a81a713d5e8a15d90c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:59:18.588436 3877 mvcc.cc:205] Tried to move back new op lower bound from 7317489084730064896 to 7317489084572987392. Current Snapshot: MvccSnapshot[applied={T|T < 7317489084730064896}]
I20260812 00:59:18.592351 3863 tablet_replica.cc:439] T f436ee197fd64fd88ca3d717be0c82f2 P e0fa056511804a81a713d5e8a15d90c1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01c6113ab1ef4026bccecc0f7633460d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } } }
I20260812 00:59:18.592705 3863 tablet_replica.cc:442] T f436ee197fd64fd88ca3d717be0c82f2 P e0fa056511804a81a713d5e8a15d90c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:59:18.596829 3867 tablet_replica.cc:439] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01c6113ab1ef4026bccecc0f7633460d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } } }
I20260812 00:59:18.597177 3867 tablet_replica.cc:442] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:59:18.598093 3794 tablet_replica.cc:439] T f436ee197fd64fd88ca3d717be0c82f2 P 8724c11a48bb4a7182aa973cd95bdb92: TxnStatusTablet state changed. Reason: New leader 01c6113ab1ef4026bccecc0f7633460d. Latest consensus state: current_term: 1 leader_uuid: "01c6113ab1ef4026bccecc0f7633460d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } } }
I20260812 00:59:18.598459 3794 tablet_replica.cc:442] T f436ee197fd64fd88ca3d717be0c82f2 P 8724c11a48bb4a7182aa973cd95bdb92: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:59:18.602703 3867 tablet_replica.cc:439] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01c6113ab1ef4026bccecc0f7633460d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } } }
I20260812 00:59:18.603058 3867 tablet_replica.cc:442] T f436ee197fd64fd88ca3d717be0c82f2 P 01c6113ab1ef4026bccecc0f7633460d: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:59:18.624243 3794 tablet_replica.cc:439] T f436ee197fd64fd88ca3d717be0c82f2 P 8724c11a48bb4a7182aa973cd95bdb92: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01c6113ab1ef4026bccecc0f7633460d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } } }
I20260812 00:59:18.624626 3794 tablet_replica.cc:442] T f436ee197fd64fd88ca3d717be0c82f2 P 8724c11a48bb4a7182aa973cd95bdb92: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:59:18.631376 3891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:18.606409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56898 (local address 127.21.201.65:34245)
0812 00:59:18.620063 (+ 13654us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:18.620069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:18.620095 (+ 26us) server_negotiation.cc:407] Connection header received
0812 00:59:18.620210 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:18.620215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:18.620325 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:18.620510 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:59:18.621525 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.622695 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.627948 (+ 5253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.628633 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.628751 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:18.629395 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:59:18.630239 (+ 844us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:59:18.630370 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:18.630897 (+ 527us) server_negotiation.cc:299] Negotiation successful
0812 00:59:18.630990 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13496,"thread_start_us":14273,"threads_started":1}
I20260812 00:59:18.651933 3780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:18.645104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:51729 (local address 127.21.201.67:33863)
0812 00:59:18.645288 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:18.645293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:18.645758 (+ 465us) server_negotiation.cc:407] Connection header received
0812 00:59:18.646041 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:18.646045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:18.646124 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:18.646219 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:59:18.646837 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.647677 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.649854 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.650348 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.650446 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:18.651057 (+ 611us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:59:18.651197 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:59:18.651280 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:18.651500 (+ 220us) server_negotiation.cc:299] Negotiation successful
0812 00:59:18.651578 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 00:59:18.652076 3899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:18.644895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:33863 (local address 127.21.201.65:51729)
0812 00:59:18.645641 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:18.645663 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:18.645825 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:18.646269 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:18.646274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:18.646288 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:59:18.646664 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:18.646677 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.647909 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.647916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:18.649691 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:18.649707 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.650552 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.650558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:18.650746 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:18.650824 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:59:18.651313 (+ 489us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:59:18.651337 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 00:59:18.651439 (+ 102us) client_negotiation.cc:240] Negotiation successful
0812 00:59:18.651596 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":393,"threads_started":1}
I20260812 00:59:18.664844 3648 raft_consensus.cc:1261] T ffacabaabc274141999ff8d008395041 P 01c6113ab1ef4026bccecc0f7633460d [term 1 FOLLOWER]: Refusing update from remote peer 8724c11a48bb4a7182aa973cd95bdb92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:59:18.665715 3385 raft_consensus.cc:1261] T ffacabaabc274141999ff8d008395041 P e0fa056511804a81a713d5e8a15d90c1 [term 1 FOLLOWER]: Refusing update from remote peer 8724c11a48bb4a7182aa973cd95bdb92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:59:18.665926 3794 consensus_queue.cc:1035] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:59:18.666471 3840 consensus_queue.cc:1035] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:59:18.675122 3882 mvcc.cc:205] Tried to move back new op lower bound from 7317489085079863296 to 7317489083809935360. Current Snapshot: MvccSnapshot[applied={T|T < 7317489085079863296}]
I20260812 00:59:18.682333 3878 mvcc.cc:205] Tried to move back new op lower bound from 7317489085079863296 to 7317489083809935360. Current Snapshot: MvccSnapshot[applied={T|T < 7317489085079863296}]
I20260812 00:59:18.712543 3891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:18.702578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:42185 (local address 127.21.201.65:34245)
0812 00:59:18.702858 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:18.702868 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:18.706101 (+ 3233us) server_negotiation.cc:407] Connection header received
0812 00:59:18.706409 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:18.706414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:18.706490 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:18.706571 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:59:18.707246 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.708209 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.710594 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.711030 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.711131 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:18.711722 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:59:18.711899 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:59:18.711983 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:18.712151 (+ 168us) server_negotiation.cc:299] Negotiation successful
0812 00:59:18.712225 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 00:59:18.712882 3908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:18.702470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:34245 (local address 127.21.201.67:42185)
0812 00:59:18.705982 (+ 3512us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:18.706004 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:18.706248 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:18.706629 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:18.706635 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:18.706648 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:59:18.707082 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:18.707095 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.708350 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.708356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:18.710444 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:18.710460 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.711236 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.711242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:18.711437 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:18.711530 (+ 93us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:59:18.712017 (+ 487us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:59:18.712031 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 00:59:18.712121 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 00:59:18.712295 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3277,"thread_start_us":3124,"threads_started":1}
I20260812 00:59:18.761890 2856 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:59:18.762205 2856 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:59:18.766825 2856 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e: Sending DeleteTablet for 3 replicas of tablet ffacabaabc274141999ff8d008395041
I20260812 00:59:18.768265 3091 tablet_service.cc:1514] Processing DeleteTablet for tablet ffacabaabc274141999ff8d008395041 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:59:18 UTC) from {username='oryx', principal='oryx/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:56896
I20260812 00:59:18.768780 3336 tablet_service.cc:1514] Processing DeleteTablet for tablet ffacabaabc274141999ff8d008395041 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:59:18 UTC) from {username='oryx', principal='oryx/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:39884
I20260812 00:59:18.769382 3913 tablet_replica.cc:330] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92: stopping tablet replica
I20260812 00:59:18.769611 3913 raft_consensus.cc:2229] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:59:18.769968 3607 tablet_service.cc:1514] Processing DeleteTablet for tablet ffacabaabc274141999ff8d008395041 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:59:18 UTC) from {username='oryx', principal='oryx/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:59738
I20260812 00:59:18.770308 3913 raft_consensus.cc:2258] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:59:18.771612 3913 ts_tablet_manager.cc:1905] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:59:18.780232 3914 tablet_replica.cc:330] T ffacabaabc274141999ff8d008395041 P e0fa056511804a81a713d5e8a15d90c1: stopping tablet replica
I20260812 00:59:18.780676 3914 raft_consensus.cc:2229] T ffacabaabc274141999ff8d008395041 P e0fa056511804a81a713d5e8a15d90c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:59:18.780819 3913 ts_tablet_manager.cc:1918] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:59:18.781024 3913 log.cc:1196] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-0/wal/wals/ffacabaabc274141999ff8d008395041
I20260812 00:59:18.781179 3914 raft_consensus.cc:2258] T ffacabaabc274141999ff8d008395041 P e0fa056511804a81a713d5e8a15d90c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:59:18.781599 3913 ts_tablet_manager.cc:1939] T ffacabaabc274141999ff8d008395041 P 8724c11a48bb4a7182aa973cd95bdb92: Deleting consensus metadata
I20260812 00:59:18.782557 3914 ts_tablet_manager.cc:1905] T ffacabaabc274141999ff8d008395041 P e0fa056511804a81a713d5e8a15d90c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:59:18.782933 2835 catalog_manager.cc:4841] TS 8724c11a48bb4a7182aa973cd95bdb92 (127.21.201.65:34245): tablet ffacabaabc274141999ff8d008395041 (table test-table [id=bdab375b68c74eb5b36c5aff39d831c2]) successfully deleted
I20260812 00:59:18.790786 3914 ts_tablet_manager.cc:1918] T ffacabaabc274141999ff8d008395041 P e0fa056511804a81a713d5e8a15d90c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:59:18.791055 3914 log.cc:1196] T ffacabaabc274141999ff8d008395041 P e0fa056511804a81a713d5e8a15d90c1: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/ffacabaabc274141999ff8d008395041
I20260812 00:59:18.791646 3914 ts_tablet_manager.cc:1939] T ffacabaabc274141999ff8d008395041 P e0fa056511804a81a713d5e8a15d90c1: Deleting consensus metadata
I20260812 00:59:18.793623 2840 catalog_manager.cc:4841] TS e0fa056511804a81a713d5e8a15d90c1 (127.21.201.66:35809): tablet ffacabaabc274141999ff8d008395041 (table test-table [id=bdab375b68c74eb5b36c5aff39d831c2]) successfully deleted
I20260812 00:59:18.796581 3923 tablet_replica.cc:330] T ffacabaabc274141999ff8d008395041 P 01c6113ab1ef4026bccecc0f7633460d: stopping tablet replica
I20260812 00:59:18.796926 3923 raft_consensus.cc:2229] T ffacabaabc274141999ff8d008395041 P 01c6113ab1ef4026bccecc0f7633460d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:59:18.797334 3923 raft_consensus.cc:2258] T ffacabaabc274141999ff8d008395041 P 01c6113ab1ef4026bccecc0f7633460d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:59:18.798843 3923 ts_tablet_manager.cc:1905] T ffacabaabc274141999ff8d008395041 P 01c6113ab1ef4026bccecc0f7633460d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:59:18.808261 3923 ts_tablet_manager.cc:1918] T ffacabaabc274141999ff8d008395041 P 01c6113ab1ef4026bccecc0f7633460d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:59:18.808539 3923 log.cc:1196] T ffacabaabc274141999ff8d008395041 P 01c6113ab1ef4026bccecc0f7633460d: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/ffacabaabc274141999ff8d008395041
I20260812 00:59:18.809100 3923 ts_tablet_manager.cc:1939] T ffacabaabc274141999ff8d008395041 P 01c6113ab1ef4026bccecc0f7633460d: Deleting consensus metadata
I20260812 00:59:18.810411 2840 catalog_manager.cc:4841] TS 01c6113ab1ef4026bccecc0f7633460d (127.21.201.67:33863): tablet ffacabaabc274141999ff8d008395041 (table test-table [id=bdab375b68c74eb5b36c5aff39d831c2]) successfully deleted
Aug 12 00:59:18 dist-test-slave-7f01 krb5kdc[2782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 00:59:18 dist-test-slave-7f01 krb5kdc[2782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:18 dist-test-slave-7f01 krb5kdc[2782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 00:59:18.872619 3851 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:18.853388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44206 (local address 127.21.201.126:42259)
0812 00:59:18.853604 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:18.853611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:18.854187 (+ 576us) server_negotiation.cc:407] Connection header received
0812 00:59:18.854388 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:18.854395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:18.854525 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:18.854648 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:18.856688 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.858098 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.860172 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.860864 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.861084 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:18.872321 (+ 11237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44206: BlockingRecv error: recv got EOF from 127.0.0.1:44206 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 12 00:59:18 dist-test-slave-7f01 krb5kdc[2782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM
I20260812 00:59:18.897706 3851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:18.876898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44218 (local address 127.21.201.126:42259)
0812 00:59:18.877096 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:18.877103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:18.877185 (+ 82us) server_negotiation.cc:407] Connection header received
0812 00:59:18.877479 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:18.877486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:18.877581 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:18.877711 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:18.879372 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.880532 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.882473 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:18.883201 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:18.883349 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:18.888966 (+ 5617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:18.888975 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:18.888980 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:18.889036 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:18.893164 (+ 4128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:18.894356 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:18.894362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:18.894367 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:18.894546 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:18.895300 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:18.895306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:18.895309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:18.895752 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:18.896109 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:18.896765 (+ 656us) server_negotiation.cc:299] Negotiation successful
0812 00:59:18.897050 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 00:59:18.902444 2856 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:59:18.903314 2856 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:59:18.912132 3091 tablet_service.cc:1467] Processing CreateTablet for tablet 23fec9ad68bb404192a1c096023500a7 (DEFAULT_TABLE table=test-table [id=96e61934440444ebbd9a224241b2c037]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:59:18.912468 3607 tablet_service.cc:1467] Processing CreateTablet for tablet 23fec9ad68bb404192a1c096023500a7 (DEFAULT_TABLE table=test-table [id=96e61934440444ebbd9a224241b2c037]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:59:18.912710 3091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23fec9ad68bb404192a1c096023500a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:18.912959 3607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23fec9ad68bb404192a1c096023500a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:18.916566 3336 tablet_service.cc:1467] Processing CreateTablet for tablet 23fec9ad68bb404192a1c096023500a7 (DEFAULT_TABLE table=test-table [id=96e61934440444ebbd9a224241b2c037]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:59:18.917045 3336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23fec9ad68bb404192a1c096023500a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:18.922400 3857 tablet_bootstrap.cc:492] T 23fec9ad68bb404192a1c096023500a7 P 8724c11a48bb4a7182aa973cd95bdb92: Bootstrap starting.
I20260812 00:59:18.924356 3857 tablet_bootstrap.cc:654] T 23fec9ad68bb404192a1c096023500a7 P 8724c11a48bb4a7182aa973cd95bdb92: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:18.932878 3853 tablet_bootstrap.cc:492] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d: Bootstrap starting.
I20260812 00:59:18.934386 3857 tablet_bootstrap.cc:492] T 23fec9ad68bb404192a1c096023500a7 P 8724c11a48bb4a7182aa973cd95bdb92: No bootstrap required, opened a new log
I20260812 00:59:18.934594 3857 ts_tablet_manager.cc:1397] T 23fec9ad68bb404192a1c096023500a7 P 8724c11a48bb4a7182aa973cd95bdb92: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20260812 00:59:18.934973 3853 tablet_bootstrap.cc:654] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:18.935614 3857 raft_consensus.cc:348] T 23fec9ad68bb404192a1c096023500a7 P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:18.936060 3857 raft_consensus.cc:374] T 23fec9ad68bb404192a1c096023500a7 P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:18.936163 3857 raft_consensus.cc:729] T 23fec9ad68bb404192a1c096023500a7 P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8724c11a48bb4a7182aa973cd95bdb92, State: Initialized, Role: FOLLOWER
I20260812 00:59:18.937871 3857 consensus_queue.cc:260] T 23fec9ad68bb404192a1c096023500a7 P 8724c11a48bb4a7182aa973cd95bdb92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:18.941025 3854 tablet_bootstrap.cc:492] T 23fec9ad68bb404192a1c096023500a7 P e0fa056511804a81a713d5e8a15d90c1: Bootstrap starting.
I20260812 00:59:18.941452 3857 ts_tablet_manager.cc:1428] T 23fec9ad68bb404192a1c096023500a7 P 8724c11a48bb4a7182aa973cd95bdb92: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20260812 00:59:18.943616 3854 tablet_bootstrap.cc:654] T 23fec9ad68bb404192a1c096023500a7 P e0fa056511804a81a713d5e8a15d90c1: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:18.961858 3854 tablet_bootstrap.cc:492] T 23fec9ad68bb404192a1c096023500a7 P e0fa056511804a81a713d5e8a15d90c1: No bootstrap required, opened a new log
I20260812 00:59:18.962119 3854 ts_tablet_manager.cc:1397] T 23fec9ad68bb404192a1c096023500a7 P e0fa056511804a81a713d5e8a15d90c1: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.001s
I20260812 00:59:18.963109 3853 tablet_bootstrap.cc:492] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d: No bootstrap required, opened a new log
I20260812 00:59:18.962972 3854 raft_consensus.cc:348] T 23fec9ad68bb404192a1c096023500a7 P e0fa056511804a81a713d5e8a15d90c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:18.963287 3854 raft_consensus.cc:374] T 23fec9ad68bb404192a1c096023500a7 P e0fa056511804a81a713d5e8a15d90c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:18.963375 3854 raft_consensus.cc:729] T 23fec9ad68bb404192a1c096023500a7 P e0fa056511804a81a713d5e8a15d90c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0fa056511804a81a713d5e8a15d90c1, State: Initialized, Role: FOLLOWER
I20260812 00:59:18.963403 3853 ts_tablet_manager.cc:1397] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.008s
I20260812 00:59:18.963886 3854 consensus_queue.cc:260] T 23fec9ad68bb404192a1c096023500a7 P e0fa056511804a81a713d5e8a15d90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:18.964392 3853 raft_consensus.cc:348] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:18.964737 3853 raft_consensus.cc:374] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:18.964859 3853 raft_consensus.cc:729] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c6113ab1ef4026bccecc0f7633460d, State: Initialized, Role: FOLLOWER
I20260812 00:59:18.965222 3853 consensus_queue.cc:260] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:18.965948 3853 ts_tablet_manager.cc:1428] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:59:18.967468 3854 ts_tablet_manager.cc:1428] T 23fec9ad68bb404192a1c096023500a7 P e0fa056511804a81a713d5e8a15d90c1: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260812 00:59:19.240309 3887 raft_consensus.cc:482] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:59:19.240650 3887 raft_consensus.cc:504] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:19.241632 3887 leader_election.cc:288] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8724c11a48bb4a7182aa973cd95bdb92 (127.21.201.65:34245), e0fa056511804a81a713d5e8a15d90c1 (127.21.201.66:35809)
I20260812 00:59:19.242429 3385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23fec9ad68bb404192a1c096023500a7" candidate_uuid: "01c6113ab1ef4026bccecc0f7633460d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0fa056511804a81a713d5e8a15d90c1" is_pre_election: true
I20260812 00:59:19.242420 3133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23fec9ad68bb404192a1c096023500a7" candidate_uuid: "01c6113ab1ef4026bccecc0f7633460d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8724c11a48bb4a7182aa973cd95bdb92" is_pre_election: true
I20260812 00:59:19.242717 3385 raft_consensus.cc:2454] T 23fec9ad68bb404192a1c096023500a7 P e0fa056511804a81a713d5e8a15d90c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01c6113ab1ef4026bccecc0f7633460d in term 0.
I20260812 00:59:19.242725 3133 raft_consensus.cc:2454] T 23fec9ad68bb404192a1c096023500a7 P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01c6113ab1ef4026bccecc0f7633460d in term 0.
I20260812 00:59:19.243268 3527 leader_election.cc:302] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01c6113ab1ef4026bccecc0f7633460d, e0fa056511804a81a713d5e8a15d90c1; no voters:
I20260812 00:59:19.243607 3887 raft_consensus.cc:2781] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:59:19.243772 3887 raft_consensus.cc:482] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:59:19.243880 3887 raft_consensus.cc:3037] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:19.247200 3887 raft_consensus.cc:504] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:19.247970 3887 leader_election.cc:288] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d [CANDIDATE]: Term 1 election: Requested vote from peers 8724c11a48bb4a7182aa973cd95bdb92 (127.21.201.65:34245), e0fa056511804a81a713d5e8a15d90c1 (127.21.201.66:35809)
I20260812 00:59:19.248580 3385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23fec9ad68bb404192a1c096023500a7" candidate_uuid: "01c6113ab1ef4026bccecc0f7633460d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0fa056511804a81a713d5e8a15d90c1"
I20260812 00:59:19.248579 3133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23fec9ad68bb404192a1c096023500a7" candidate_uuid: "01c6113ab1ef4026bccecc0f7633460d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8724c11a48bb4a7182aa973cd95bdb92"
I20260812 00:59:19.248821 3385 raft_consensus.cc:3037] T 23fec9ad68bb404192a1c096023500a7 P e0fa056511804a81a713d5e8a15d90c1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:19.248858 3133 raft_consensus.cc:3037] T 23fec9ad68bb404192a1c096023500a7 P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:19.250962 3385 raft_consensus.cc:2454] T 23fec9ad68bb404192a1c096023500a7 P e0fa056511804a81a713d5e8a15d90c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01c6113ab1ef4026bccecc0f7633460d in term 1.
I20260812 00:59:19.251470 3527 leader_election.cc:302] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01c6113ab1ef4026bccecc0f7633460d, e0fa056511804a81a713d5e8a15d90c1; no voters:
I20260812 00:59:19.251864 3887 raft_consensus.cc:2781] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:59:19.251905 3133 raft_consensus.cc:2454] T 23fec9ad68bb404192a1c096023500a7 P 8724c11a48bb4a7182aa973cd95bdb92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01c6113ab1ef4026bccecc0f7633460d in term 1.
I20260812 00:59:19.252540 3887 raft_consensus.cc:686] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d [term 1 LEADER]: Becoming Leader. State: Replica: 01c6113ab1ef4026bccecc0f7633460d, State: Running, Role: LEADER
I20260812 00:59:19.252853 3887 consensus_queue.cc:237] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:19.255882 2857 catalog_manager.cc:5495] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d reported cstate change: term changed from 0 to 1, leader changed from <none> to 01c6113ab1ef4026bccecc0f7633460d (127.21.201.67). New cstate: current_term: 1 leader_uuid: "01c6113ab1ef4026bccecc0f7633460d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } health_report { overall_health: HEALTHY } } }
I20260812 00:59:19.302022 3962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:19.296467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59772 (local address 127.21.201.67:33863)
0812 00:59:19.297008 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:19.297020 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:19.297034 (+ 14us) server_negotiation.cc:407] Connection header received
0812 00:59:19.297091 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:19.297095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:19.297161 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:19.297275 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:59:19.297673 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:19.298415 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:19.300164 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:19.300609 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:19.300758 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:19.301250 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:59:19.301359 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:59:19.301475 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:19.301620 (+ 145us) server_negotiation.cc:299] Negotiation successful
0812 00:59:19.301704 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":269,"threads_started":1}
I20260812 00:59:19.305359 3133 raft_consensus.cc:1261] T 23fec9ad68bb404192a1c096023500a7 P 8724c11a48bb4a7182aa973cd95bdb92 [term 1 FOLLOWER]: Refusing update from remote peer 01c6113ab1ef4026bccecc0f7633460d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:59:19.305337 3385 raft_consensus.cc:1261] T 23fec9ad68bb404192a1c096023500a7 P e0fa056511804a81a713d5e8a15d90c1 [term 1 FOLLOWER]: Refusing update from remote peer 01c6113ab1ef4026bccecc0f7633460d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:59:19.305987 3887 consensus_queue.cc:1035] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:59:19.306296 3958 consensus_queue.cc:1035] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:59:19.320364 2855 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:59:19.320693 2855 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:59:19.325964 2855 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e: Sending DeleteTablet for 3 replicas of tablet 23fec9ad68bb404192a1c096023500a7
I20260812 00:59:19.326936 3091 tablet_service.cc:1514] Processing DeleteTablet for tablet 23fec9ad68bb404192a1c096023500a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:59:19 UTC) from {username='oryx', principal='oryx/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:56896
I20260812 00:59:19.326963 3088 tablet_service.cc:1514] Processing DeleteTablet for tablet 23fec9ad68bb404192a1c096023500a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:59:19 UTC) from {username='oryx', principal='oryx/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:56896
I20260812 00:59:19.327696 3978 tablet_replica.cc:330] T 23fec9ad68bb404192a1c096023500a7 P 8724c11a48bb4a7182aa973cd95bdb92: stopping tablet replica
I20260812 00:59:19.327685 3607 tablet_service.cc:1514] Processing DeleteTablet for tablet 23fec9ad68bb404192a1c096023500a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:59:19 UTC) from {username='oryx', principal='oryx/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:59738
I20260812 00:59:19.327940 3978 raft_consensus.cc:2229] T 23fec9ad68bb404192a1c096023500a7 P 8724c11a48bb4a7182aa973cd95bdb92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:59:19.328068 3606 tablet_service.cc:1514] Processing DeleteTablet for tablet 23fec9ad68bb404192a1c096023500a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:59:19 UTC) from {username='oryx', principal='oryx/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:59738
I20260812 00:59:19.328156 3336 tablet_service.cc:1514] Processing DeleteTablet for tablet 23fec9ad68bb404192a1c096023500a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:59:19 UTC) from {username='oryx', principal='oryx/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:39884
I20260812 00:59:19.328240 3978 raft_consensus.cc:2258] T 23fec9ad68bb404192a1c096023500a7 P 8724c11a48bb4a7182aa973cd95bdb92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:59:19.328766 3981 tablet_replica.cc:330] T 23fec9ad68bb404192a1c096023500a7 P e0fa056511804a81a713d5e8a15d90c1: stopping tablet replica
I20260812 00:59:19.329030 3981 raft_consensus.cc:2229] T 23fec9ad68bb404192a1c096023500a7 P e0fa056511804a81a713d5e8a15d90c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:59:19.328907 2853 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 00:59:19.329357 3981 raft_consensus.cc:2258] T 23fec9ad68bb404192a1c096023500a7 P e0fa056511804a81a713d5e8a15d90c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:59:19.329416 3978 ts_tablet_manager.cc:1905] T 23fec9ad68bb404192a1c096023500a7 P 8724c11a48bb4a7182aa973cd95bdb92: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 00:59:19.329742 2853 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:59:19.331808 3980 tablet_replica.cc:330] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d: stopping tablet replica
I20260812 00:59:19.332026 3980 raft_consensus.cc:2229] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:59:19.332407 3980 raft_consensus.cc:2258] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:59:19.332705 3981 ts_tablet_manager.cc:1905] T 23fec9ad68bb404192a1c096023500a7 P e0fa056511804a81a713d5e8a15d90c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:59:19.333537 3980 ts_tablet_manager.cc:1905] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:59:19.339629 3336 tablet_service.cc:1467] Processing CreateTablet for tablet a686b49d1d74455aa74ac45ae7b2611b (DEFAULT_TABLE table=test-table [id=7ff65fd5b1a24401913623c43aa127c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:59:19.340126 3336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a686b49d1d74455aa74ac45ae7b2611b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:19.340029 3607 tablet_service.cc:1467] Processing CreateTablet for tablet a686b49d1d74455aa74ac45ae7b2611b (DEFAULT_TABLE table=test-table [id=7ff65fd5b1a24401913623c43aa127c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:59:19.340534 3607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a686b49d1d74455aa74ac45ae7b2611b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:19.346076 3978 ts_tablet_manager.cc:1918] T 23fec9ad68bb404192a1c096023500a7 P 8724c11a48bb4a7182aa973cd95bdb92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:59:19.346223 3978 log.cc:1196] T 23fec9ad68bb404192a1c096023500a7 P 8724c11a48bb4a7182aa973cd95bdb92: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-0/wal/wals/23fec9ad68bb404192a1c096023500a7
I20260812 00:59:19.346794 3978 ts_tablet_manager.cc:1939] T 23fec9ad68bb404192a1c096023500a7 P 8724c11a48bb4a7182aa973cd95bdb92: Deleting consensus metadata
I20260812 00:59:19.346884 3854 tablet_bootstrap.cc:492] T a686b49d1d74455aa74ac45ae7b2611b P e0fa056511804a81a713d5e8a15d90c1: Bootstrap starting.
I20260812 00:59:19.347434 3853 tablet_bootstrap.cc:492] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d: Bootstrap starting.
I20260812 00:59:19.347826 3091 tablet_service.cc:1467] Processing CreateTablet for tablet a686b49d1d74455aa74ac45ae7b2611b (DEFAULT_TABLE table=test-table [id=7ff65fd5b1a24401913623c43aa127c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:59:19.348297 3091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a686b49d1d74455aa74ac45ae7b2611b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:19.348717 3854 tablet_bootstrap.cc:654] T a686b49d1d74455aa74ac45ae7b2611b P e0fa056511804a81a713d5e8a15d90c1: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:19.349117 3853 tablet_bootstrap.cc:654] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:19.352260 2835 catalog_manager.cc:4841] TS 8724c11a48bb4a7182aa973cd95bdb92 (127.21.201.65:34245): tablet 23fec9ad68bb404192a1c096023500a7 (table test-table [id=96e61934440444ebbd9a224241b2c037]) successfully deleted
I20260812 00:59:19.339598 3981 ts_tablet_manager.cc:1918] T 23fec9ad68bb404192a1c096023500a7 P e0fa056511804a81a713d5e8a15d90c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:59:19.353449 3981 log.cc:1196] T 23fec9ad68bb404192a1c096023500a7 P e0fa056511804a81a713d5e8a15d90c1: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/23fec9ad68bb404192a1c096023500a7
W20260812 00:59:19.353834 2835 catalog_manager.cc:4808] TS 8724c11a48bb4a7182aa973cd95bdb92 (127.21.201.65:34245): delete failed for tablet 23fec9ad68bb404192a1c096023500a7 because the tablet was not found. No further retry: Not found: Tablet not found: 23fec9ad68bb404192a1c096023500a7
I20260812 00:59:19.353966 3981 ts_tablet_manager.cc:1939] T 23fec9ad68bb404192a1c096023500a7 P e0fa056511804a81a713d5e8a15d90c1: Deleting consensus metadata
I20260812 00:59:19.355186 2840 catalog_manager.cc:4841] TS e0fa056511804a81a713d5e8a15d90c1 (127.21.201.66:35809): tablet 23fec9ad68bb404192a1c096023500a7 (table test-table [id=96e61934440444ebbd9a224241b2c037]) successfully deleted
I20260812 00:59:19.355296 3980 ts_tablet_manager.cc:1918] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:59:19.355479 3980 log.cc:1196] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/23fec9ad68bb404192a1c096023500a7
I20260812 00:59:19.356606 3980 ts_tablet_manager.cc:1939] T 23fec9ad68bb404192a1c096023500a7 P 01c6113ab1ef4026bccecc0f7633460d: Deleting consensus metadata
I20260812 00:59:19.357864 2840 catalog_manager.cc:4841] TS 01c6113ab1ef4026bccecc0f7633460d (127.21.201.67:33863): tablet 23fec9ad68bb404192a1c096023500a7 (table test-table [id=96e61934440444ebbd9a224241b2c037]) successfully deleted
W20260812 00:59:19.358503 2840 catalog_manager.cc:4808] TS 01c6113ab1ef4026bccecc0f7633460d (127.21.201.67:33863): delete failed for tablet 23fec9ad68bb404192a1c096023500a7 because the tablet was not found. No further retry: Not found: Tablet not found: 23fec9ad68bb404192a1c096023500a7
I20260812 00:59:19.359951 3857 tablet_bootstrap.cc:492] T a686b49d1d74455aa74ac45ae7b2611b P 8724c11a48bb4a7182aa973cd95bdb92: Bootstrap starting.
I20260812 00:59:19.361680 3854 tablet_bootstrap.cc:492] T a686b49d1d74455aa74ac45ae7b2611b P e0fa056511804a81a713d5e8a15d90c1: No bootstrap required, opened a new log
I20260812 00:59:19.361882 3854 ts_tablet_manager.cc:1397] T a686b49d1d74455aa74ac45ae7b2611b P e0fa056511804a81a713d5e8a15d90c1: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.006s
I20260812 00:59:19.361990 3857 tablet_bootstrap.cc:654] T a686b49d1d74455aa74ac45ae7b2611b P 8724c11a48bb4a7182aa973cd95bdb92: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:19.362700 3854 raft_consensus.cc:348] T a686b49d1d74455aa74ac45ae7b2611b P e0fa056511804a81a713d5e8a15d90c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:19.362952 3854 raft_consensus.cc:374] T a686b49d1d74455aa74ac45ae7b2611b P e0fa056511804a81a713d5e8a15d90c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:19.363037 3854 raft_consensus.cc:729] T a686b49d1d74455aa74ac45ae7b2611b P e0fa056511804a81a713d5e8a15d90c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0fa056511804a81a713d5e8a15d90c1, State: Initialized, Role: FOLLOWER
I20260812 00:59:19.363286 3854 consensus_queue.cc:260] T a686b49d1d74455aa74ac45ae7b2611b P e0fa056511804a81a713d5e8a15d90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:19.364444 3853 tablet_bootstrap.cc:492] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d: No bootstrap required, opened a new log
I20260812 00:59:19.364449 3857 tablet_bootstrap.cc:492] T a686b49d1d74455aa74ac45ae7b2611b P 8724c11a48bb4a7182aa973cd95bdb92: No bootstrap required, opened a new log
I20260812 00:59:19.364610 3853 ts_tablet_manager.cc:1397] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.003s
I20260812 00:59:19.364666 3857 ts_tablet_manager.cc:1397] T a686b49d1d74455aa74ac45ae7b2611b P 8724c11a48bb4a7182aa973cd95bdb92: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260812 00:59:19.365489 3853 raft_consensus.cc:348] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:19.365782 3853 raft_consensus.cc:374] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:19.365890 3853 raft_consensus.cc:729] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c6113ab1ef4026bccecc0f7633460d, State: Initialized, Role: FOLLOWER
I20260812 00:59:19.365697 3857 raft_consensus.cc:348] T a686b49d1d74455aa74ac45ae7b2611b P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:19.366272 3857 raft_consensus.cc:374] T a686b49d1d74455aa74ac45ae7b2611b P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:19.366133 3853 consensus_queue.cc:260] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:19.366441 3857 raft_consensus.cc:729] T a686b49d1d74455aa74ac45ae7b2611b P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8724c11a48bb4a7182aa973cd95bdb92, State: Initialized, Role: FOLLOWER
I20260812 00:59:19.366758 3853 ts_tablet_manager.cc:1428] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260812 00:59:19.366801 3857 consensus_queue.cc:260] T a686b49d1d74455aa74ac45ae7b2611b P 8724c11a48bb4a7182aa973cd95bdb92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:19.367522 3857 ts_tablet_manager.cc:1428] T a686b49d1d74455aa74ac45ae7b2611b P 8724c11a48bb4a7182aa973cd95bdb92: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260812 00:59:19.369501 3854 ts_tablet_manager.cc:1428] T a686b49d1d74455aa74ac45ae7b2611b P e0fa056511804a81a713d5e8a15d90c1: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20260812 00:59:19.390789 3887 raft_consensus.cc:482] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:59:19.391136 3887 raft_consensus.cc:504] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:19.391964 3887 leader_election.cc:288] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8724c11a48bb4a7182aa973cd95bdb92 (127.21.201.65:34245), e0fa056511804a81a713d5e8a15d90c1 (127.21.201.66:35809)
I20260812 00:59:19.392656 3133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a686b49d1d74455aa74ac45ae7b2611b" candidate_uuid: "01c6113ab1ef4026bccecc0f7633460d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8724c11a48bb4a7182aa973cd95bdb92" is_pre_election: true
I20260812 00:59:19.392679 3385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a686b49d1d74455aa74ac45ae7b2611b" candidate_uuid: "01c6113ab1ef4026bccecc0f7633460d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0fa056511804a81a713d5e8a15d90c1" is_pre_election: true
I20260812 00:59:19.392967 3385 raft_consensus.cc:2454] T a686b49d1d74455aa74ac45ae7b2611b P e0fa056511804a81a713d5e8a15d90c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01c6113ab1ef4026bccecc0f7633460d in term 0.
I20260812 00:59:19.392975 3133 raft_consensus.cc:2454] T a686b49d1d74455aa74ac45ae7b2611b P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01c6113ab1ef4026bccecc0f7633460d in term 0.
I20260812 00:59:19.393469 3527 leader_election.cc:302] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01c6113ab1ef4026bccecc0f7633460d, e0fa056511804a81a713d5e8a15d90c1; no voters:
I20260812 00:59:19.393776 3887 raft_consensus.cc:2781] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:59:19.393886 3887 raft_consensus.cc:482] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:59:19.394033 3887 raft_consensus.cc:3037] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:19.396749 3887 raft_consensus.cc:504] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:19.397457 3887 leader_election.cc:288] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d [CANDIDATE]: Term 1 election: Requested vote from peers 8724c11a48bb4a7182aa973cd95bdb92 (127.21.201.65:34245), e0fa056511804a81a713d5e8a15d90c1 (127.21.201.66:35809)
I20260812 00:59:19.397989 3133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a686b49d1d74455aa74ac45ae7b2611b" candidate_uuid: "01c6113ab1ef4026bccecc0f7633460d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8724c11a48bb4a7182aa973cd95bdb92"
I20260812 00:59:19.398078 3385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a686b49d1d74455aa74ac45ae7b2611b" candidate_uuid: "01c6113ab1ef4026bccecc0f7633460d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0fa056511804a81a713d5e8a15d90c1"
I20260812 00:59:19.398285 3133 raft_consensus.cc:3037] T a686b49d1d74455aa74ac45ae7b2611b P 8724c11a48bb4a7182aa973cd95bdb92 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:19.398299 3385 raft_consensus.cc:3037] T a686b49d1d74455aa74ac45ae7b2611b P e0fa056511804a81a713d5e8a15d90c1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:19.400892 3133 raft_consensus.cc:2454] T a686b49d1d74455aa74ac45ae7b2611b P 8724c11a48bb4a7182aa973cd95bdb92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01c6113ab1ef4026bccecc0f7633460d in term 1.
I20260812 00:59:19.401209 3385 raft_consensus.cc:2454] T a686b49d1d74455aa74ac45ae7b2611b P e0fa056511804a81a713d5e8a15d90c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01c6113ab1ef4026bccecc0f7633460d in term 1.
I20260812 00:59:19.401443 3523 leader_election.cc:302] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01c6113ab1ef4026bccecc0f7633460d, 8724c11a48bb4a7182aa973cd95bdb92; no voters:
I20260812 00:59:19.401844 3887 raft_consensus.cc:2781] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:59:19.402046 3887 raft_consensus.cc:686] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d [term 1 LEADER]: Becoming Leader. State: Replica: 01c6113ab1ef4026bccecc0f7633460d, State: Running, Role: LEADER
I20260812 00:59:19.402365 3887 consensus_queue.cc:237] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } }
I20260812 00:59:19.405181 2853 catalog_manager.cc:5495] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d reported cstate change: term changed from 0 to 1, leader changed from <none> to 01c6113ab1ef4026bccecc0f7633460d (127.21.201.67). New cstate: current_term: 1 leader_uuid: "01c6113ab1ef4026bccecc0f7633460d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01c6113ab1ef4026bccecc0f7633460d" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33863 } health_report { overall_health: HEALTHY } } }
I20260812 00:59:19.438970 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:19.433618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:33863 (local address 127.21.201.67:46687)
0812 00:59:19.434167 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:19.434183 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:19.434313 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:19.434592 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:19.434595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:19.434604 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:59:19.434842 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:19.434850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:19.435978 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:19.435984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:19.437117 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:19.437129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:19.437935 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:19.437939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:19.438062 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:19.438099 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:59:19.438466 (+ 367us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:59:19.438474 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 00:59:19.438541 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 00:59:19.438666 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":3,"thread_start_us":306,"threads_started":1}
I20260812 00:59:19.438968 3962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:19.433545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:46687 (local address 127.21.201.67:33863)
0812 00:59:19.433729 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:19.433734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:19.434270 (+ 536us) server_negotiation.cc:407] Connection header received
0812 00:59:19.434416 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:19.434420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:19.434487 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:19.434562 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:59:19.435026 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:19.435853 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:19.437270 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:19.437761 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:19.437868 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:19.438266 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:59:19.438370 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:59:19.438452 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:19.438581 (+ 129us) server_negotiation.cc:299] Negotiation successful
0812 00:59:19.438675 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":68}
I20260812 00:59:19.447499 3133 raft_consensus.cc:1261] T a686b49d1d74455aa74ac45ae7b2611b P 8724c11a48bb4a7182aa973cd95bdb92 [term 1 FOLLOWER]: Refusing update from remote peer 01c6113ab1ef4026bccecc0f7633460d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:59:19.447580 3385 raft_consensus.cc:1261] T a686b49d1d74455aa74ac45ae7b2611b P e0fa056511804a81a713d5e8a15d90c1 [term 1 FOLLOWER]: Refusing update from remote peer 01c6113ab1ef4026bccecc0f7633460d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:59:19.448122 3958 consensus_queue.cc:1035] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8724c11a48bb4a7182aa973cd95bdb92" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:59:19.448457 3887 consensus_queue.cc:1035] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0fa056511804a81a713d5e8a15d90c1" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:59:19.494906 2855 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:59:19.495167 2855 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:59:19.498310 2855 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e09f1a6ea1e47e6b18e9fe50fe2cb8e: Sending DeleteTablet for 3 replicas of tablet a686b49d1d74455aa74ac45ae7b2611b
I20260812 00:59:19.499197 3091 tablet_service.cc:1514] Processing DeleteTablet for tablet a686b49d1d74455aa74ac45ae7b2611b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:59:19 UTC) from {username='oryx', principal='oryx/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:56896
I20260812 00:59:19.499464 3336 tablet_service.cc:1514] Processing DeleteTablet for tablet a686b49d1d74455aa74ac45ae7b2611b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:59:19 UTC) from {username='oryx', principal='oryx/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:39884
I20260812 00:59:19.499558 3978 tablet_replica.cc:330] T a686b49d1d74455aa74ac45ae7b2611b P 8724c11a48bb4a7182aa973cd95bdb92: stopping tablet replica
I20260812 00:59:19.499828 3981 tablet_replica.cc:330] T a686b49d1d74455aa74ac45ae7b2611b P e0fa056511804a81a713d5e8a15d90c1: stopping tablet replica
I20260812 00:59:19.499814 3978 raft_consensus.cc:2229] T a686b49d1d74455aa74ac45ae7b2611b P 8724c11a48bb4a7182aa973cd95bdb92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:59:19.500036 3978 raft_consensus.cc:2258] T a686b49d1d74455aa74ac45ae7b2611b P 8724c11a48bb4a7182aa973cd95bdb92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:59:19.500070 3981 raft_consensus.cc:2229] T a686b49d1d74455aa74ac45ae7b2611b P e0fa056511804a81a713d5e8a15d90c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:59:19.500407 3607 tablet_service.cc:1514] Processing DeleteTablet for tablet a686b49d1d74455aa74ac45ae7b2611b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:59:19 UTC) from {username='oryx', principal='oryx/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:59738
I20260812 00:59:19.500658 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 2988
I20260812 00:59:19.500742 3981 raft_consensus.cc:2258] T a686b49d1d74455aa74ac45ae7b2611b P e0fa056511804a81a713d5e8a15d90c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:59:19.500931 3980 tablet_replica.cc:330] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d: stopping tablet replica
I20260812 00:59:19.501145 3980 raft_consensus.cc:2229] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:59:19.501510 3980 raft_consensus.cc:2258] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:59:19.502012 3981 ts_tablet_manager.cc:1905] T a686b49d1d74455aa74ac45ae7b2611b P e0fa056511804a81a713d5e8a15d90c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:59:19.503103 3978 ts_tablet_manager.cc:1905] T a686b49d1d74455aa74ac45ae7b2611b P 8724c11a48bb4a7182aa973cd95bdb92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:59:19.504130 3980 ts_tablet_manager.cc:1905] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:59:19.510506 3978 ts_tablet_manager.cc:1918] T a686b49d1d74455aa74ac45ae7b2611b P 8724c11a48bb4a7182aa973cd95bdb92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:59:19.510675 3978 log.cc:1196] T a686b49d1d74455aa74ac45ae7b2611b P 8724c11a48bb4a7182aa973cd95bdb92: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-0/wal/wals/a686b49d1d74455aa74ac45ae7b2611b
I20260812 00:59:19.511070 3980 ts_tablet_manager.cc:1918] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:59:19.511148 3978 ts_tablet_manager.cc:1939] T a686b49d1d74455aa74ac45ae7b2611b P 8724c11a48bb4a7182aa973cd95bdb92: Deleting consensus metadata
I20260812 00:59:19.511233 3980 log.cc:1196] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/a686b49d1d74455aa74ac45ae7b2611b
I20260812 00:59:19.511231 3981 ts_tablet_manager.cc:1918] T a686b49d1d74455aa74ac45ae7b2611b P e0fa056511804a81a713d5e8a15d90c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:59:19.511443 3981 log.cc:1196] T a686b49d1d74455aa74ac45ae7b2611b P e0fa056511804a81a713d5e8a15d90c1: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/a686b49d1d74455aa74ac45ae7b2611b
I20260812 00:59:19.511853 3980 ts_tablet_manager.cc:1939] T a686b49d1d74455aa74ac45ae7b2611b P 01c6113ab1ef4026bccecc0f7633460d: Deleting consensus metadata
I20260812 00:59:19.511973 3981 ts_tablet_manager.cc:1939] T a686b49d1d74455aa74ac45ae7b2611b P e0fa056511804a81a713d5e8a15d90c1: Deleting consensus metadata
I20260812 00:59:19.513235 2840 catalog_manager.cc:4841] TS e0fa056511804a81a713d5e8a15d90c1 (127.21.201.66:35809): tablet a686b49d1d74455aa74ac45ae7b2611b (table test-table [id=7ff65fd5b1a24401913623c43aa127c8]) successfully deleted
I20260812 00:59:19.513219 2835 catalog_manager.cc:4841] TS 8724c11a48bb4a7182aa973cd95bdb92 (127.21.201.65:34245): tablet a686b49d1d74455aa74ac45ae7b2611b (table test-table [id=7ff65fd5b1a24401913623c43aa127c8]) successfully deleted
I20260812 00:59:19.513636 2840 catalog_manager.cc:4841] TS 01c6113ab1ef4026bccecc0f7633460d (127.21.201.67:33863): tablet a686b49d1d74455aa74ac45ae7b2611b (table test-table [id=7ff65fd5b1a24401913623c43aa127c8]) successfully deleted
Aug 12 00:59:19 dist-test-slave-7f01 krb5kdc[2782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.201.65@KRBTEST.COM
I20260812 00:59:19.528998 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:19.502313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56904 (local address 127.21.201.65:34245)
0812 00:59:19.513116 (+ 10803us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:19.513124 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:19.513149 (+ 25us) server_negotiation.cc:407] Connection header received
0812 00:59:19.513215 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:19.513221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:19.513328 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:19.513540 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:19.515135 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:19.515987 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:19.517305 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:19.517755 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:19.517868 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:19.522895 (+ 5027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:19.522914 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:19.522932 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:19.522973 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:19.525978 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:19.526758 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:19.526775 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:19.526777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:19.526922 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:19.527343 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:19.527346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:19.527348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:19.527733 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:19.528035 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:19.528379 (+ 344us) server_negotiation.cc:299] Negotiation successful
0812 00:59:19.528587 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10639,"thread_start_us":10503,"threads_started":1}
I20260812 00:59:19.529762 3202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:19.668779 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 2988
I20260812 00:59:19.699985 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 3220
Aug 12 00:59:19 dist-test-slave-7f01 krb5kdc[2782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.201.66@KRBTEST.COM
I20260812 00:59:19.717180 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:19.700886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39900 (local address 127.21.201.66:35809)
0812 00:59:19.701424 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:19.701429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:19.701446 (+ 17us) server_negotiation.cc:407] Connection header received
0812 00:59:19.701507 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:19.701511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:19.701571 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:19.701692 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:19.703098 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:19.704051 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:19.705273 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:19.705732 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:19.705839 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:19.710433 (+ 4594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:19.710451 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:19.710467 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:19.710504 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:19.713699 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:19.714838 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:19.714848 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:19.714851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:19.715001 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:19.715432 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:19.715437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:19.715440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:19.715913 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:19.716235 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:19.716540 (+ 305us) server_negotiation.cc:299] Negotiation successful
0812 00:59:19.716807 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":291,"threads_started":1}
I20260812 00:59:19.718134 3474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:19.822053 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 3220
I20260812 00:59:19.850165 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 3486
Aug 12 00:59:19 dist-test-slave-7f01 krb5kdc[2782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.201.67@KRBTEST.COM
I20260812 00:59:19.867600 3962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:19.851077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59782 (local address 127.21.201.67:33863)
0812 00:59:19.851304 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:19.851309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:19.851328 (+ 19us) server_negotiation.cc:407] Connection header received
0812 00:59:19.851388 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:19.851392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:19.851467 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:19.851583 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:19.852974 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:19.853786 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:19.855203 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:19.855637 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:19.855898 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:19.860610 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:19.860636 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:19.860653 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:19.860691 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:19.864089 (+ 3398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:19.864899 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:19.864906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:19.864909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:19.865063 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:19.865484 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:19.865488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:19.865490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:19.865965 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:19.866208 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:19.866890 (+ 682us) server_negotiation.cc:299] Negotiation successful
0812 00:59:19.867148 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260812 00:59:19.868413 3725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:20.034020 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 3486
I20260812 00:59:20.075294 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 2804
I20260812 00:59:20.077224 2931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:20.211362 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 2804
2026-08-12T00:59:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6812 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:59:20 dist-test-slave-7f01 krb5kdc[4020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:59:20 dist-test-slave-7f01 krb5kdc[4020](info): set up 2 sockets
Aug 12 00:59:20 dist-test-slave-7f01 krb5kdc[4020](info): commencing operation
krb5kdc: starting...
W20260812 00:59:21.612398 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.346s user 0.014s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 00:59:21 dist-test-slave-7f01 krb5kdc[4020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:59:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:59:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 00:59:21.907809 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:40587
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:40587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:59:22.260673 4042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:22.260991 4042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:22.261173 4042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:22.268729 4042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:59:22.268860 4042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:22.268977 4042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:22.269061 4042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:59:22.269150 4042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:59:22.279762 4042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:40587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:40587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:22.281195 4042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:22.282574 4042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:22.295179 4052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:22.295246 4057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:22.295295 4051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:22.296491 4042 server_base.cc:1034] running on GCE node
I20260812 00:59:22.297343 4042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:22.298786 4042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:22.300012 4042 hybrid_clock.cc:648] HybridClock initialized: now 1786496362299958 us; error 45 us; skew 500 ppm
I20260812 00:59:22.300348 4042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-12T00:59:22Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2150 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:59:22 dist-test-slave-7f01 krb5kdc[4074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:59:22 dist-test-slave-7f01 krb5kdc[4074](info): set up 2 sockets
Aug 12 00:59:22 dist-test-slave-7f01 krb5kdc[4074](info): commencing operation
krb5kdc: starting...
W20260812 00:59:23.747244 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.330s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 00:59:23 dist-test-slave-7f01 krb5kdc[4074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:59:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:59:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 00:59:24.032085 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:38217
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:40823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:38217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:59:24.389487 4095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:24.389829 4095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:24.390026 4095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:24.397565 4095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:59:24.397692 4095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:24.397740 4095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:24.397874 4095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:59:24.397948 4095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:59:24.409255 4095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:40823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:38217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:38217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:24.410696 4095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:24.412400 4095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:24.425016 4103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:24.425043 4106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:24.425211 4102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:24.425944 4095 server_base.cc:1034] running on GCE node
I20260812 00:59:24.426717 4095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:24.428330 4095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:24.429576 4095 hybrid_clock.cc:648] HybridClock initialized: now 1786496364429504 us; error 56 us; skew 500 ppm
I20260812 00:59:24.429942 4095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:24 dist-test-slave-7f01 krb5kdc[4074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496364, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:24.438320 4095 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 00:59:24.440415 4095 webserver.cc:460] Webserver started at http://127.21.201.126:33277/ using document root <none> and password file <none>
I20260812 00:59:24.441203 4095 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:24.441316 4095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:24.441593 4095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:59:24.445415 4095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "c0d4aee122fb45688fadfd6886638211"
format_stamp: "Formatted at 2026-08-12 00:59:24 on dist-test-slave-7f01"
server_key: "9028c2c54e12510640c048bea784d108"
server_key_iv: "c87ab70fda86dbfc74362e3ff47e29c8"
server_key_version: "encryptionkey@0"
I20260812 00:59:24.446168 4095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "c0d4aee122fb45688fadfd6886638211"
format_stamp: "Formatted at 2026-08-12 00:59:24 on dist-test-slave-7f01"
server_key: "9028c2c54e12510640c048bea784d108"
server_key_iv: "c87ab70fda86dbfc74362e3ff47e29c8"
server_key_version: "encryptionkey@0"
I20260812 00:59:24.451882 4095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 00:59:24.456029 4116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:24.457892 4095 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 00:59:24.458137 4095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "c0d4aee122fb45688fadfd6886638211"
format_stamp: "Formatted at 2026-08-12 00:59:24 on dist-test-slave-7f01"
server_key: "9028c2c54e12510640c048bea784d108"
server_key_iv: "c87ab70fda86dbfc74362e3ff47e29c8"
server_key_version: "encryptionkey@0"
I20260812 00:59:24.458381 4095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:24.482003 4095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:24.487099 4095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:24.487468 4095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:24.509409 4095 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:38217
I20260812 00:59:24.509445 4198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:38217 every 8 connection(s)
I20260812 00:59:24.510957 4095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 00:59:24.514909 4199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:24.521279 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 4095
I20260812 00:59:24.521517 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 00:59:24.522372 22309 external_mini_cluster.cc:1428] Setting key ba02e8ef64387b2c6aea62948daefb22
I20260812 00:59:24.527959 4199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211: Bootstrap starting.
I20260812 00:59:24.532020 4199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:24.533679 4199 log.cc:824] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211: Log is configured to *not* fsync() on all Append() calls
I20260812 00:59:24.537732 4199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211: No bootstrap required, opened a new log
Aug 12 00:59:24 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:59:24.546433 4199 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d4aee122fb45688fadfd6886638211" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38217 } }
I20260812 00:59:24.546947 4199 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:24.547096 4199 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d4aee122fb45688fadfd6886638211, State: Initialized, Role: FOLLOWER
I20260812 00:59:24.547930 4199 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d4aee122fb45688fadfd6886638211" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38217 } }
I20260812 00:59:24.548243 4199 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:59:24.548403 4199 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:59:24.548619 4199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:24.549554 4204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:24.524685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46514 (local address 127.21.201.126:38217)
0812 00:59:24.525659 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:24.525671 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:24.525720 (+ 49us) server_negotiation.cc:407] Connection header received
0812 00:59:24.527563 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:24.527585 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:24.528061 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:24.528448 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:24.530656 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:24.532578 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:24.534817 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:24.535835 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:24.535995 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:24.541093 (+ 5098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:24.541101 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:24.541117 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:24.541164 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:24.544633 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:24.545415 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:24.545422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:24.545430 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:24.545623 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:24.546042 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:24.546048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:24.546053 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:24.546620 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:24.546981 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:24.547485 (+ 504us) server_negotiation.cc:299] Negotiation successful
0812 00:59:24.547999 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":479,"threads_started":1}
I20260812 00:59:24.551476 4199 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d4aee122fb45688fadfd6886638211" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38217 } }
I20260812 00:59:24.552100 4199 leader_election.cc:302] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0d4aee122fb45688fadfd6886638211; no voters:
I20260812 00:59:24.552819 4199 leader_election.cc:288] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:59:24.552981 4206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:59:24.553732 4206 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [term 1 LEADER]: Becoming Leader. State: Replica: c0d4aee122fb45688fadfd6886638211, State: Running, Role: LEADER
I20260812 00:59:24.554395 4206 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d4aee122fb45688fadfd6886638211" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38217 } }
I20260812 00:59:24.555246 4199 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:59:24.557899 4207 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0d4aee122fb45688fadfd6886638211" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d4aee122fb45688fadfd6886638211" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38217 } } }
I20260812 00:59:24.557871 4209 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0d4aee122fb45688fadfd6886638211. Latest consensus state: current_term: 1 leader_uuid: "c0d4aee122fb45688fadfd6886638211" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d4aee122fb45688fadfd6886638211" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38217 } } }
I20260812 00:59:24.558468 4209 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [sys.catalog]: This master's current role is: LEADER
I20260812 00:59:24.558481 4207 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [sys.catalog]: This master's current role is: LEADER
I20260812 00:59:24.560256 4214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:59:24.566502 4214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:59:24.576575 4214 catalog_manager.cc:1347] Generated new cluster ID: b8c25d05373c46ac8fd600995b90389f
I20260812 00:59:24.576766 4214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:59:24.603974 4214 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:59:24.605564 4214 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:59:24.611843 4214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211: Generated new TSK 0
I20260812 00:59:24.612872 4214 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.201.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
I20260812 00:59:24.684348 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:0
--local_ip_for_outbound_sockets=127.21.201.65
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38217
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:40823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:59:25.076783 4244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:25.077143 4244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:25.077251 4244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:25.087105 4244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:25.087265 4244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:25.087404 4244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 00:59:25.099519 4244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:40823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:25.101163 4244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:25.102537 4244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:25.118371 4253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:25.118458 4252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:25.118418 4257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:25.119568 4244 server_base.cc:1034] running on GCE node
I20260812 00:59:25.120296 4244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:25.121513 4244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:25.122833 4244 hybrid_clock.cc:648] HybridClock initialized: now 1786496365122772 us; error 50 us; skew 500 ppm
I20260812 00:59:25.123273 4244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:25 dist-test-slave-7f01 krb5kdc[4074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496365, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:25.127395 4244 init.cc:377] Logged in from keytab as kudu/127.21.201.65@KRBTEST.COM (short username kudu)
I20260812 00:59:25.129431 4244 webserver.cc:460] Webserver started at http://127.21.201.65:42543/ using document root <none> and password file <none>
I20260812 00:59:25.130209 4244 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:25.130367 4244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:25.130715 4244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:59:25.134172 4244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/data/instance:
uuid: "7e69d26b3d9e47519326d5089a3746bb"
format_stamp: "Formatted at 2026-08-12 00:59:25 on dist-test-slave-7f01"
server_key: "78ac7d08141878d243751fc9b9851716"
server_key_iv: "0449581dffbed54447e6520c9dd5066c"
server_key_version: "encryptionkey@0"
I20260812 00:59:25.135731 4244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance:
uuid: "7e69d26b3d9e47519326d5089a3746bb"
format_stamp: "Formatted at 2026-08-12 00:59:25 on dist-test-slave-7f01"
server_key: "78ac7d08141878d243751fc9b9851716"
server_key_iv: "0449581dffbed54447e6520c9dd5066c"
server_key_version: "encryptionkey@0"
I20260812 00:59:25.142719 4244 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 00:59:25.146843 4266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:25.148835 4244 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 00:59:25.149052 4244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "7e69d26b3d9e47519326d5089a3746bb"
format_stamp: "Formatted at 2026-08-12 00:59:25 on dist-test-slave-7f01"
server_key: "78ac7d08141878d243751fc9b9851716"
server_key_iv: "0449581dffbed54447e6520c9dd5066c"
server_key_version: "encryptionkey@0"
I20260812 00:59:25.149304 4244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:25.188856 4244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:25.193650 4244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:25.193979 4244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:25.195170 4244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:59:25.197041 4244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:59:25.197155 4244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:25.197335 4244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:59:25.197417 4244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:25.242908 4244 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:38357
I20260812 00:59:25.243027 4471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:38357 every 8 connection(s)
I20260812 00:59:25.244433 4244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 00:59:25.248211 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 4244
I20260812 00:59:25.248442 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance
I20260812 00:59:25.249171 22309 external_mini_cluster.cc:1428] Setting key 528657223e3252f8695f35e393af3d3c
Aug 12 00:59:25 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496365, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:59:25.269611 4477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:25.247106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:42973 (local address 127.21.201.126:38217)
0812 00:59:25.247583 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:25.247587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:25.248680 (+ 1093us) server_negotiation.cc:407] Connection header received
0812 00:59:25.250663 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:25.250668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:25.250743 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:25.250868 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:25.252561 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:25.253843 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:25.255582 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:25.256358 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:25.256513 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:25.261362 (+ 4849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:25.261372 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:25.261379 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:25.261429 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:25.265492 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:25.266633 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:25.266637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:25.266639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:25.266817 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:25.267390 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:25.267394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:25.267396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:25.267724 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:25.267931 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:25.269047 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0812 00:59:25.269293 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":259,"threads_started":1}
I20260812 00:59:25.272084 4479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:25.247569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.65:42973)
0812 00:59:25.248487 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:25.248556 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:25.250376 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:25.251134 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:25.251143 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:25.251703 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:25.252371 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:25.252389 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:25.254045 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:25.254050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:25.255430 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:25.255443 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:25.256657 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:25.256668 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:25.256920 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:25.257707 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:25.257735 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:25.261085 (+ 3350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:25.265681 (+ 4596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:25.265694 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:25.265716 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:25.266455 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:25.266961 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:25.266969 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:25.266974 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:25.267252 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:25.267933 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:25.267944 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:25.268518 (+ 574us) client_negotiation.cc:769] Sending connection context
0812 00:59:25.268876 (+ 358us) client_negotiation.cc:240] Negotiation successful
0812 00:59:25.269942 (+ 1066us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":449,"threads_started":1}
I20260812 00:59:25.273931 4472 heartbeater.cc:347] Connected to a master server at 127.21.201.126:38217
I20260812 00:59:25.274403 4472 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:25.275257 4472 heartbeater.cc:511] Master 127.21.201.126:38217 requested a full tablet report, sending...
I20260812 00:59:25.278364 4145 ts_manager.cc:194] Registered new tserver with Master: 7e69d26b3d9e47519326d5089a3746bb (127.21.201.65:38357)
I20260812 00:59:25.280109 4145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.65@KRBTEST.COM'} at 127.21.201.65:42973
WARNING: no policy specified for kudu/127.21.201.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
I20260812 00:59:25.324424 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:0
--local_ip_for_outbound_sockets=127.21.201.66
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38217
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:40823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:59:25.703081 4483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:25.703599 4483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:25.703853 4483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:25.712577 4483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:25.712707 4483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:25.712945 4483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 00:59:25.723282 4483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:40823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:25.725343 4483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:25.727239 4483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:25.741953 4491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:25.741866 4494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:25.741909 4497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:25.743125 4483 server_base.cc:1034] running on GCE node
I20260812 00:59:25.743913 4483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:25.745373 4483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:25.746642 4483 hybrid_clock.cc:648] HybridClock initialized: now 1786496365746579 us; error 51 us; skew 500 ppm
I20260812 00:59:25.747037 4483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:25 dist-test-slave-7f01 krb5kdc[4074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496365, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:25.750479 4483 init.cc:377] Logged in from keytab as kudu/127.21.201.66@KRBTEST.COM (short username kudu)
I20260812 00:59:25.752075 4483 webserver.cc:460] Webserver started at http://127.21.201.66:35215/ using document root <none> and password file <none>
I20260812 00:59:25.752743 4483 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:25.752876 4483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:25.753247 4483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:59:25.756556 4483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/data/instance:
uuid: "8ec2984e65a640e19f71e84c5249898f"
format_stamp: "Formatted at 2026-08-12 00:59:25 on dist-test-slave-7f01"
server_key: "35e0c4d29b8d17b28a967a08d6aeca45"
server_key_iv: "d95bfe88eafde05906a527860b0723bd"
server_key_version: "encryptionkey@0"
I20260812 00:59:25.757431 4483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance:
uuid: "8ec2984e65a640e19f71e84c5249898f"
format_stamp: "Formatted at 2026-08-12 00:59:25 on dist-test-slave-7f01"
server_key: "35e0c4d29b8d17b28a967a08d6aeca45"
server_key_iv: "d95bfe88eafde05906a527860b0723bd"
server_key_version: "encryptionkey@0"
I20260812 00:59:25.763666 4483 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 00:59:25.767875 4514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:25.769361 4483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 00:59:25.769572 4483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "8ec2984e65a640e19f71e84c5249898f"
format_stamp: "Formatted at 2026-08-12 00:59:25 on dist-test-slave-7f01"
server_key: "35e0c4d29b8d17b28a967a08d6aeca45"
server_key_iv: "d95bfe88eafde05906a527860b0723bd"
server_key_version: "encryptionkey@0"
I20260812 00:59:25.769786 4483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:25.788611 4483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:25.793804 4483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:25.794153 4483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:25.795209 4483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:59:25.796677 4483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:59:25.796754 4483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:25.796942 4483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:59:25.796998 4483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:25.833591 4483 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:38357
I20260812 00:59:25.833771 4693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:38357 every 8 connection(s)
I20260812 00:59:25.834949 4483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 00:59:25.840137 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 4483
I20260812 00:59:25.840334 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance
I20260812 00:59:25.840818 22309 external_mini_cluster.cc:1428] Setting key 1fcaeef8b1a73d98a0bc5022fc84e06f
Aug 12 00:59:25 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496365, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:59:25.856431 4701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:25.837652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:60825 (local address 127.21.201.126:38217)
0812 00:59:25.838182 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:25.838187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:25.839119 (+ 932us) server_negotiation.cc:407] Connection header received
0812 00:59:25.841260 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:25.841266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:25.841362 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:25.841494 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:25.843082 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:25.843901 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:25.845462 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:25.845912 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:25.846048 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:25.849894 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:25.849900 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:25.849904 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:25.849939 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:25.852937 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:25.853786 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:25.853792 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:25.853796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:25.853972 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:25.854484 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:25.854489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:25.854492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:25.854804 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:25.854965 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:25.855768 (+ 803us) server_negotiation.cc:299] Negotiation successful
0812 00:59:25.856007 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":284,"threads_started":1}
I20260812 00:59:25.857455 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:25.838229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.66:60825)
0812 00:59:25.838954 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:25.838999 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:25.840989 (+ 1990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:25.841726 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:25.841734 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:25.842289 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:25.842911 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:25.842927 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:25.844066 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:25.844070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:25.845307 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:25.845319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:25.846161 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:25.846167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:25.846515 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:25.847245 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:25.847270 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:25.849627 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:25.853107 (+ 3480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:25.853118 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:25.853138 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:25.853664 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:25.854088 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:25.854093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:25.854097 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:25.854353 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:25.854938 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:25.854946 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:25.855360 (+ 414us) client_negotiation.cc:769] Sending connection context
0812 00:59:25.855669 (+ 309us) client_negotiation.cc:240] Negotiation successful
0812 00:59:25.856258 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":280,"threads_started":1}
I20260812 00:59:25.859050 4695 heartbeater.cc:347] Connected to a master server at 127.21.201.126:38217
I20260812 00:59:25.859434 4695 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:25.860423 4695 heartbeater.cc:511] Master 127.21.201.126:38217 requested a full tablet report, sending...
I20260812 00:59:25.862950 4145 ts_manager.cc:194] Registered new tserver with Master: 8ec2984e65a640e19f71e84c5249898f (127.21.201.66:38357)
I20260812 00:59:25.864331 4145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.66@KRBTEST.COM'} at 127.21.201.66:60825
WARNING: no policy specified for kudu/127.21.201.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
I20260812 00:59:25.911275 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:0
--local_ip_for_outbound_sockets=127.21.201.67
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38217
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:40823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:59:26.250559 4464 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.21.201.65:38357
I20260812 00:59:26.283725 4472 heartbeater.cc:503] Master 127.21.201.126:38217 was elected leader, sending a full tablet report...
W20260812 00:59:26.378765 4710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:26.379455 4710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:26.379843 4710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:26.394345 4710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:26.394655 4710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:26.395072 4710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 00:59:26.415515 4710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:40823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:26.418241 4710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:26.421037 4710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:26.448853 4719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:26.449088 4721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:26.451726 4718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:26.453783 4710 server_base.cc:1034] running on GCE node
I20260812 00:59:26.455657 4710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:26.458612 4710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:26.464100 4710 hybrid_clock.cc:648] HybridClock initialized: now 1786496366463940 us; error 166 us; skew 500 ppm
I20260812 00:59:26.465014 4710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:26 dist-test-slave-7f01 krb5kdc[4074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496366, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:26.474067 4710 init.cc:377] Logged in from keytab as kudu/127.21.201.67@KRBTEST.COM (short username kudu)
I20260812 00:59:26.479015 4710 webserver.cc:460] Webserver started at http://127.21.201.67:45789/ using document root <none> and password file <none>
I20260812 00:59:26.480588 4710 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:26.480968 4710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:26.482163 4710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:59:26.491380 4710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/data/instance:
uuid: "eae6b34c07624bd08815ddda40717a9a"
format_stamp: "Formatted at 2026-08-12 00:59:26 on dist-test-slave-7f01"
server_key: "51da14d39370369faa66d11213f70a28"
server_key_iv: "36a8afaf8f1c4de7eec02a2eaf1b8e65"
server_key_version: "encryptionkey@0"
I20260812 00:59:26.493742 4710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance:
uuid: "eae6b34c07624bd08815ddda40717a9a"
format_stamp: "Formatted at 2026-08-12 00:59:26 on dist-test-slave-7f01"
server_key: "51da14d39370369faa66d11213f70a28"
server_key_iv: "36a8afaf8f1c4de7eec02a2eaf1b8e65"
server_key_version: "encryptionkey@0"
I20260812 00:59:26.509225 4710 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.010s sys 0.005s
I20260812 00:59:26.522377 4731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:26.525976 4710 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.001s
I20260812 00:59:26.526314 4710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "eae6b34c07624bd08815ddda40717a9a"
format_stamp: "Formatted at 2026-08-12 00:59:26 on dist-test-slave-7f01"
server_key: "51da14d39370369faa66d11213f70a28"
server_key_iv: "36a8afaf8f1c4de7eec02a2eaf1b8e65"
server_key_version: "encryptionkey@0"
I20260812 00:59:26.526821 4710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:26.579051 4710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:26.587410 4710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:26.587940 4710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:26.589721 4710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:59:26.592308 4710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:59:26.592465 4710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:26.592609 4710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:59:26.592700 4710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:26.660724 4710 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:38443
I20260812 00:59:26.660918 4927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:38443 every 8 connection(s)
I20260812 00:59:26.663357 4710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 00:59:26.668751 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 4710
I20260812 00:59:26.669301 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance
I20260812 00:59:26.670193 22309 external_mini_cluster.cc:1428] Setting key 7bf03ef9b95a1cb5804cfb3839dd2002
Aug 12 00:59:26 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496366, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:59:26.704943 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:26.668484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:60407 (local address 127.21.201.126:38217)
0812 00:59:26.669575 (+ 1091us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:26.669586 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:26.671341 (+ 1755us) server_negotiation.cc:407] Connection header received
0812 00:59:26.674787 (+ 3446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:26.674794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:26.674968 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:26.675199 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:26.678103 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:26.679436 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:26.682195 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:26.682942 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:26.683280 (+ 338us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:26.690331 (+ 7051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:26.690338 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:26.690341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:26.690414 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:26.696322 (+ 5908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:26.698104 (+ 1782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:26.698111 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:26.698114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:26.698405 (+ 291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:26.699313 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:26.699319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:26.699322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:26.700139 (+ 817us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:26.700380 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:26.703229 (+ 2849us) server_negotiation.cc:299] Negotiation successful
0812 00:59:26.703872 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":553,"threads_started":1}
I20260812 00:59:26.707262 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:26.669618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.67:60407)
0812 00:59:26.671088 (+ 1470us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:26.671152 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:26.674327 (+ 3175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:26.675627 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:26.675647 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:26.676767 (+ 1120us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:26.677838 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:26.677862 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:26.679684 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:26.679698 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:26.681938 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:26.681968 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:26.683506 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:26.683521 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:26.684042 (+ 521us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:26.685184 (+ 1142us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:26.685208 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:26.689894 (+ 4686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:26.696706 (+ 6812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:26.696723 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:26.696748 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:26.697753 (+ 1005us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:26.698614 (+ 861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:26.698620 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:26.698623 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:26.699019 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:26.700721 (+ 1702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:26.700753 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:26.702129 (+ 1376us) client_negotiation.cc:769] Sending connection context
0812 00:59:26.702906 (+ 777us) client_negotiation.cc:240] Negotiation successful
0812 00:59:26.704352 (+ 1446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"thread_start_us":575,"threads_started":1}
I20260812 00:59:26.709849 4929 heartbeater.cc:347] Connected to a master server at 127.21.201.126:38217
I20260812 00:59:26.710417 4929 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:26.711789 4929 heartbeater.cc:511] Master 127.21.201.126:38217 requested a full tablet report, sending...
I20260812 00:59:26.714550 4145 ts_manager.cc:194] Registered new tserver with Master: eae6b34c07624bd08815ddda40717a9a (127.21.201.67:38443)
I20260812 00:59:26.716675 4145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.67@KRBTEST.COM'} at 127.21.201.67:60407
I20260812 00:59:26.718632 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.201.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.201.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.201.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.65.keytab.
Entry for principal oryx/127.21.201.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.65.keytab.
W20260812 00:59:26.822957 22309 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.21.201.65: real 0.101s user 0.005s sys 0.034s
W20260812 00:59:26.841629 4689 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.21.201.66:38357
WARNING: no policy specified for oryx/127.21.201.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.201.66@KRBTEST.COM" created.
I20260812 00:59:26.868577 4695 heartbeater.cc:503] Master 127.21.201.126:38217 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.201.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.66.keytab.
Entry for principal oryx/127.21.201.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.66.keytab.
WARNING: no policy specified for oryx/127.21.201.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.201.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.201.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.67.keytab.
Entry for principal oryx/127.21.201.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.67.keytab.
W20260812 00:59:27.021811 22309 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.21.201.67: real 0.101s user 0.000s sys 0.034s
I20260812 00:59:27.022439 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 4244
Aug 12 00:59:27 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.65@KRBTEST.COM
I20260812 00:59:27.057049 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:27.025554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46748 (local address 127.21.201.65:38357)
0812 00:59:27.027074 (+ 1520us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:27.027084 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:27.027127 (+ 43us) server_negotiation.cc:407] Connection header received
0812 00:59:27.027228 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:27.027244 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:27.027583 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:27.027961 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:27.030740 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:27.032629 (+ 1889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:27.034825 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:27.036005 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:27.036168 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:27.045187 (+ 9019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:27.045225 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:27.045253 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:27.045329 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:27.051138 (+ 5809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:27.052743 (+ 1605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:27.052753 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:27.052756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:27.053096 (+ 340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:27.053746 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:27.053750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:27.053752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:27.054440 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:27.055016 (+ 576us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:27.055556 (+ 540us) server_negotiation.cc:299] Negotiation successful
0812 00:59:27.056190 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1182,"thread_start_us":653,"threads_started":1}
I20260812 00:59:27.059437 4463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:27.189725 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 4244
I20260812 00:59:27.240661 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 4483
Aug 12 00:59:27 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.66@KRBTEST.COM
I20260812 00:59:27.274302 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:27.242713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52602 (local address 127.21.201.66:38357)
0812 00:59:27.243944 (+ 1231us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:27.243962 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:27.244016 (+ 54us) server_negotiation.cc:407] Connection header received
0812 00:59:27.244124 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:27.244132 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:27.244519 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:27.244723 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:27.247320 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:27.249208 (+ 1888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:27.251435 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:27.252730 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:27.252909 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:27.261339 (+ 8430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:27.261366 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:27.261402 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:27.261491 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:27.267193 (+ 5702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:27.268932 (+ 1739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:27.268952 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:27.268954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:27.269291 (+ 337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:27.270187 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:27.270200 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:27.270203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:27.270985 (+ 782us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:27.271433 (+ 448us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:27.272526 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0812 00:59:27.273506 (+ 980us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":561,"threads_started":1}
I20260812 00:59:27.277213 4688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:27.402422 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 4483
I20260812 00:59:27.451284 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 4710
Aug 12 00:59:27 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.67@KRBTEST.COM
I20260812 00:59:27.481082 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:27.453068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41220 (local address 127.21.201.67:38443)
0812 00:59:27.453982 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:27.453989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:27.454050 (+ 61us) server_negotiation.cc:407] Connection header received
0812 00:59:27.454168 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:27.454184 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:27.454615 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:27.454834 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:27.457422 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:27.458958 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:27.461464 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:27.462692 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:27.462874 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:27.470326 (+ 7452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:27.470350 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:27.470384 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:27.470455 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:27.475127 (+ 4672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:27.476368 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:27.476376 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:27.476379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:27.476619 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:27.477252 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:27.477258 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:27.477261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:27.477768 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:27.478190 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:27.479196 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0812 00:59:27.480151 (+ 955us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":461,"threads_started":1}
I20260812 00:59:27.483356 4920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:27.575853 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 4710
I20260812 00:59:27.601399 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 4095
I20260812 00:59:27.603185 4192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:27.714664 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 4095
I20260812 00:59:27.748489 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:38217
--webserver_interface=127.21.201.126
--webserver_port=33277
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:40823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:38217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:59:28.110816 4969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:28.111166 4969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:28.111269 4969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:28.118837 4969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:59:28.118973 4969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:28.119025 4969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:28.119086 4969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:59:28.119131 4969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:59:28.129451 4969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:40823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:38217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:38217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=33277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:28.130853 4969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:28.132169 4969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:28.144925 4978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:28.145004 4982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:28.145027 4979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:28.145664 4969 server_base.cc:1034] running on GCE node
I20260812 00:59:28.146596 4969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:28.148236 4969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:28.149490 4969 hybrid_clock.cc:648] HybridClock initialized: now 1786496368149433 us; error 68 us; skew 500 ppm
I20260812 00:59:28.149878 4969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:28 dist-test-slave-7f01 krb5kdc[4074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496368, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:28.153298 4969 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 00:59:28.155223 4969 webserver.cc:460] Webserver started at http://127.21.201.126:33277/ using document root <none> and password file <none>
I20260812 00:59:28.156051 4969 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:28.156222 4969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:28.162153 4969 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 00:59:28.165815 4997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:28.167534 4969 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260812 00:59:28.167824 4969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "c0d4aee122fb45688fadfd6886638211"
format_stamp: "Formatted at 2026-08-12 00:59:24 on dist-test-slave-7f01"
server_key: "9028c2c54e12510640c048bea784d108"
server_key_iv: "c87ab70fda86dbfc74362e3ff47e29c8"
server_key_version: "encryptionkey@0"
I20260812 00:59:28.168926 4969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:28.193739 4969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:28.198676 4969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:28.198979 4969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:28.221519 4969 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:38217
I20260812 00:59:28.221532 5091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:38217 every 8 connection(s)
I20260812 00:59:28.223102 4969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 00:59:28.230084 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 4969
I20260812 00:59:28.230152 5092 sys_catalog.cc:263] Verifying existing consensus state
I20260812 00:59:28.230970 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:38357
--local_ip_for_outbound_sockets=127.21.201.65
--tserver_master_addrs=127.21.201.126:38217
--webserver_port=42543
--webserver_interface=127.21.201.65
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:40823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.201.65
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:59:28.233664 5092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211: Bootstrap starting.
I20260812 00:59:28.259399 5092 log.cc:824] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211: Log is configured to *not* fsync() on all Append() calls
I20260812 00:59:28.272614 5092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:59:28.273375 5092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211: Bootstrap complete.
I20260812 00:59:28.287927 5092 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d4aee122fb45688fadfd6886638211" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38217 } }
I20260812 00:59:28.288857 5092 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d4aee122fb45688fadfd6886638211, State: Initialized, Role: FOLLOWER
I20260812 00:59:28.289512 5092 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d4aee122fb45688fadfd6886638211" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38217 } }
I20260812 00:59:28.289774 5092 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:59:28.289930 5092 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:59:28.290129 5092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [term 1 FOLLOWER]: Advancing to term 2
I20260812 00:59:28.294234 5092 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d4aee122fb45688fadfd6886638211" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38217 } }
I20260812 00:59:28.294770 5092 leader_election.cc:302] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0d4aee122fb45688fadfd6886638211; no voters:
I20260812 00:59:28.295470 5092 leader_election.cc:288] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 00:59:28.295629 5098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 00:59:28.296867 5098 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [term 2 LEADER]: Becoming Leader. State: Replica: c0d4aee122fb45688fadfd6886638211, State: Running, Role: LEADER
I20260812 00:59:28.297545 5098 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d4aee122fb45688fadfd6886638211" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38217 } }
I20260812 00:59:28.297902 5092 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:59:28.300149 5101 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0d4aee122fb45688fadfd6886638211. Latest consensus state: current_term: 2 leader_uuid: "c0d4aee122fb45688fadfd6886638211" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d4aee122fb45688fadfd6886638211" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38217 } } }
I20260812 00:59:28.300611 5101 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [sys.catalog]: This master's current role is: LEADER
I20260812 00:59:28.301275 5100 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c0d4aee122fb45688fadfd6886638211" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d4aee122fb45688fadfd6886638211" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38217 } } }
I20260812 00:59:28.301651 5100 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211 [sys.catalog]: This master's current role is: LEADER
I20260812 00:59:28.305415 5110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:59:28.312736 5110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:59:28.314643 5110 catalog_manager.cc:1259] Loaded cluster ID: b8c25d05373c46ac8fd600995b90389f
I20260812 00:59:28.314855 5110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:59:28.320143 5110 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:59:28.322317 5110 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c0d4aee122fb45688fadfd6886638211: Loaded TSK: 0
I20260812 00:59:28.324445 5110 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 00:59:28.636673 5096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:28.637012 5096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:28.637173 5096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:28.644984 5096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:28.645116 5096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:28.645298 5096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 00:59:28.656663 5096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:40823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.65.keytab
--principal=oryx/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:38357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=42543
--tserver_master_addrs=127.21.201.126:38217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:28.658217 5096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:28.659839 5096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:28.675114 5143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:28.675366 5141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:28.675611 5145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:28.677175 5096 server_base.cc:1034] running on GCE node
I20260812 00:59:28.678069 5096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:28.679380 5096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:28.680646 5096 hybrid_clock.cc:648] HybridClock initialized: now 1786496368680587 us; error 47 us; skew 500 ppm
I20260812 00:59:28.681033 5096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:28 dist-test-slave-7f01 krb5kdc[4074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496368, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:28.685518 5096 init.cc:377] Logged in from keytab as oryx/127.21.201.65@KRBTEST.COM (short username oryx)
I20260812 00:59:28.687288 5096 webserver.cc:460] Webserver started at http://127.21.201.65:42543/ using document root <none> and password file <none>
I20260812 00:59:28.688161 5096 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:28.688321 5096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:28.694411 5096 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 00:59:28.697417 5162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:28.699158 5096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 00:59:28.699352 5096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "7e69d26b3d9e47519326d5089a3746bb"
format_stamp: "Formatted at 2026-08-12 00:59:25 on dist-test-slave-7f01"
server_key: "78ac7d08141878d243751fc9b9851716"
server_key_iv: "0449581dffbed54447e6520c9dd5066c"
server_key_version: "encryptionkey@0"
I20260812 00:59:28.700466 5096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:28.714004 5096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:28.718204 5096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:28.718514 5096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:28.719549 5096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:59:28.721065 5096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:59:28.721159 5096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:28.721321 5096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:59:28.721427 5096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:28.756521 5096 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:38357
I20260812 00:59:28.756619 5376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:38357 every 8 connection(s)
I20260812 00:59:28.758037 5096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 00:59:28.762001 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 5096
I20260812 00:59:28.762832 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:38357
--local_ip_for_outbound_sockets=127.21.201.66
--tserver_master_addrs=127.21.201.126:38217
--webserver_port=35215
--webserver_interface=127.21.201.66
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:40823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.201.66
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 00:59:28 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:28 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:28.777518 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:28.761382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.65:44959)
0812 00:59:28.762435 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:28.762472 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:28.764412 (+ 1940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:28.765779 (+ 1367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:28.765793 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:28.766448 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:28.766986 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:28.767004 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:28.768740 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:28.768745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:28.769977 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:28.769989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:28.771022 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:28.771030 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:28.771312 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:28.772140 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:28.772164 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:28.776481 (+ 4317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":561,"threads_started":1}
W20260812 00:59:28.778141 5381 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:28.761296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:44959 (local address 127.21.201.126:38217)
0812 00:59:28.762082 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:28.762178 (+ 96us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:28.762597 (+ 419us) server_negotiation.cc:407] Connection header received
0812 00:59:28.764966 (+ 2369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:28.764976 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:28.765291 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:28.765647 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:28.767203 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:28.768559 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:28.770132 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:28.770763 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:28.770978 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:28.776996 (+ 6018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.65:44959: BlockingRecv error: recv got EOF from 127.21.201.65:44959 (error 108)
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":278,"threads_started":1}
W20260812 00:59:28.778440 5377 heartbeater.cc:644] Failed to heartbeat to 127.21.201.126:38217 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.201.126:38217: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Aug 12 00:59:28 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:28 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:28.788297 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:28.779170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.65:50133)
0812 00:59:28.779358 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:28.779375 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:28.779472 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:28.779856 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:28.779859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:28.780190 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:28.780339 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:28.780347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:28.781906 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:28.781910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:28.782844 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:28.782855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:28.783946 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:28.783950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:28.784095 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:28.784773 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:28.784791 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:28.788071 (+ 3280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 00:59:28.788697 5381 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:28.779323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:50133 (local address 127.21.201.126:38217)
0812 00:59:28.779518 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:28.779526 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:28.779547 (+ 21us) server_negotiation.cc:407] Connection header received
0812 00:59:28.779613 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:28.779621 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:28.779706 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:28.779835 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:28.780512 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:28.781724 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:28.782998 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:28.783777 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:28.783922 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:28.788397 (+ 4475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.65:50133: BlockingRecv error: recv got EOF from 127.21.201.65:50133 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 12 00:59:28 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:28 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:28.797930 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:28.789182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.65:60769)
0812 00:59:28.789336 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:28.789361 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:28.789449 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:28.789823 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:28.789826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:28.790113 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:28.790262 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:28.790269 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:28.791894 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:28.791898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:28.792835 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:28.792845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:28.793933 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:28.793937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:28.794090 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:28.794713 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:28.794732 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:28.797575 (+ 2843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260812 00:59:28.798157 5381 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:28.789290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:60769 (local address 127.21.201.126:38217)
0812 00:59:28.789485 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:28.789493 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:28.789516 (+ 23us) server_negotiation.cc:407] Connection header received
0812 00:59:28.789579 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:28.789586 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:28.789686 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:28.789808 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:28.790410 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:28.791718 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:28.792987 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:28.793778 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:28.793952 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:28.797886 (+ 3934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.65:60769: BlockingRecv error: recv got EOF from 127.21.201.65:60769 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260812 00:59:28.798516 5377 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 00:59:29.124316 5385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:29.124641 5385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:29.124815 5385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:29.132465 5385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:29.132599 5385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:29.132792 5385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 00:59:29.143349 5385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:40823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.66.keytab
--principal=oryx/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:38357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=35215
--tserver_master_addrs=127.21.201.126:38217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:29.144948 5385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:29.146313 5385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:29.160065 5394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:29.160164 5392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:29.160190 5396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:29.161146 5385 server_base.cc:1034] running on GCE node
I20260812 00:59:29.161857 5385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:29.162943 5385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:29.164184 5385 hybrid_clock.cc:648] HybridClock initialized: now 1786496369164149 us; error 30 us; skew 500 ppm
I20260812 00:59:29.164552 5385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:29 dist-test-slave-7f01 krb5kdc[4074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496369, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:29.169018 5385 init.cc:377] Logged in from keytab as oryx/127.21.201.66@KRBTEST.COM (short username oryx)
I20260812 00:59:29.170837 5385 webserver.cc:460] Webserver started at http://127.21.201.66:35215/ using document root <none> and password file <none>
I20260812 00:59:29.171556 5385 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:29.171679 5385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:29.177814 5385 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 00:59:29.180702 5407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:29.182420 5385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 00:59:29.182583 5385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "8ec2984e65a640e19f71e84c5249898f"
format_stamp: "Formatted at 2026-08-12 00:59:25 on dist-test-slave-7f01"
server_key: "35e0c4d29b8d17b28a967a08d6aeca45"
server_key_iv: "d95bfe88eafde05906a527860b0723bd"
server_key_version: "encryptionkey@0"
I20260812 00:59:29.183600 5385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:29.209437 5385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:29.214206 5385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:29.214470 5385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:29.215520 5385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:59:29.217124 5385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:59:29.217226 5385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:29.217316 5385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:59:29.217383 5385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:29.255955 5385 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:38357
I20260812 00:59:29.256026 5624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:38357 every 8 connection(s)
I20260812 00:59:29.257592 5385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 00:59:29.266512 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 5385
I20260812 00:59:29.267231 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:38443
--local_ip_for_outbound_sockets=127.21.201.67
--tserver_master_addrs=127.21.201.126:38217
--webserver_port=45789
--webserver_interface=127.21.201.67
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:40823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.201.67
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 00:59:29 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:29 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 00:59:29.275074 5381 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:29.260638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:59771 (local address 127.21.201.126:38217)
0812 00:59:29.260839 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:29.260844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:29.262098 (+ 1254us) server_negotiation.cc:407] Connection header received
0812 00:59:29.264220 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:29.264226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:29.264330 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:29.264453 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:29.266060 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.266863 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.268585 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.269049 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.269188 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:29.274790 (+ 5602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.66:59771: BlockingRecv error: recv got EOF from 127.21.201.66:59771 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 00:59:29.275950 5632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:29.261129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.66:59771)
0812 00:59:29.261879 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:29.261937 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:29.263922 (+ 1985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:29.264702 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:29.264711 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:29.265277 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:29.265862 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:29.265878 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.267037 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.267041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:29.268437 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:29.268450 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.269326 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.269335 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:29.269564 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:29.270308 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:29.270332 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:29.274589 (+ 4257us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":304,"threads_started":1}
W20260812 00:59:29.276659 5625 heartbeater.cc:644] Failed to heartbeat to 127.21.201.126:38217 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.201.126:38217: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Aug 12 00:59:29 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:29 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:29.286278 5632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:29.277296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.66:52773)
0812 00:59:29.277491 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:29.277510 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:29.277611 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:29.278041 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:29.278045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:29.278423 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:29.278578 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:29.278586 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.279873 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.279877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:29.280825 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:29.280836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.281572 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.281578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:29.281760 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:29.282399 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:29.282417 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:29.285973 (+ 3556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260812 00:59:29.287159 5381 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:29.277417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:52773 (local address 127.21.201.126:38217)
0812 00:59:29.277640 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:29.277648 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:29.277674 (+ 26us) server_negotiation.cc:407] Connection header received
0812 00:59:29.277748 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:29.277755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:29.277845 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:29.277979 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:29.278745 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.279647 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.280980 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.281417 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.281556 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:29.286888 (+ 5332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.66:52773: BlockingRecv error: recv got EOF from 127.21.201.66:52773 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 12 00:59:29 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:29 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:29.297472 5632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:29.287074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.66:55223)
0812 00:59:29.287531 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:29.287548 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:29.287691 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:29.288272 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:29.288278 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:29.288620 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:29.288837 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:29.288846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.290412 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.290416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:29.291722 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:29.291851 (+ 129us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.293120 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.293125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:29.293283 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:29.293980 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:29.293998 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:29.297211 (+ 3213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":324}
W20260812 00:59:29.297868 5381 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:29.287426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:55223 (local address 127.21.201.126:38217)
0812 00:59:29.287645 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:29.287653 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:29.287677 (+ 24us) server_negotiation.cc:407] Connection header received
0812 00:59:29.287840 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:29.287847 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:29.287944 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:29.288237 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:29.288993 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.290244 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.292074 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.292902 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.293070 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:29.297569 (+ 4499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.66:55223: BlockingRecv error: recv got EOF from 127.21.201.66:55223 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260812 00:59:29.297863 5625 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 00:59:29.631220 5633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:29.631560 5633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:29.631757 5633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:29.639964 5633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:29.640100 5633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:29.640291 5633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 00:59:29.650810 5633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:40823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/krb5kdc/oryx_127.21.201.67.keytab
--principal=oryx/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:38443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=45789
--tserver_master_addrs=127.21.201.126:38217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:29.652343 5633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:29.653708 5633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:29.667870 5644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:29.667898 5649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:29.667821 5647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:29.668851 5633 server_base.cc:1034] running on GCE node
I20260812 00:59:29.669572 5633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:29.671730 5633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:29.673032 5633 hybrid_clock.cc:648] HybridClock initialized: now 1786496369672968 us; error 57 us; skew 500 ppm
I20260812 00:59:29.673463 5633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:29 dist-test-slave-7f01 krb5kdc[4074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496369, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:29.680598 5633 init.cc:377] Logged in from keytab as oryx/127.21.201.67@KRBTEST.COM (short username oryx)
I20260812 00:59:29.682257 5633 webserver.cc:460] Webserver started at http://127.21.201.67:45789/ using document root <none> and password file <none>
I20260812 00:59:29.682984 5633 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:29.683110 5633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:29.689038 5633 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 00:59:29.692102 5664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:29.693560 5633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 00:59:29.693760 5633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "eae6b34c07624bd08815ddda40717a9a"
format_stamp: "Formatted at 2026-08-12 00:59:26 on dist-test-slave-7f01"
server_key: "51da14d39370369faa66d11213f70a28"
server_key_iv: "36a8afaf8f1c4de7eec02a2eaf1b8e65"
server_key_version: "encryptionkey@0"
I20260812 00:59:29.694821 5633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:29.709362 5633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:29.714277 5633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:29.714612 5633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:29.715693 5633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:59:29.717233 5633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:59:29.717376 5633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:29.717499 5633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:59:29.717567 5633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:29.759481 5633 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:38443
I20260812 00:59:29.759539 5873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:38443 every 8 connection(s)
I20260812 00:59:29.761247 5633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
W20260812 00:59:29.762367 5362 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.21.201.65:38357
I20260812 00:59:29.771783 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 5633
Aug 12 00:59:29 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:29 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 00:59:29.778613 5381 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:29.764165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:48889 (local address 127.21.201.126:38217)
0812 00:59:29.764417 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:29.764422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:29.765566 (+ 1144us) server_negotiation.cc:407] Connection header received
0812 00:59:29.767449 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:29.767453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:29.767532 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:29.767627 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:29.769243 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.770005 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.771625 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.772167 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.772304 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:29.778305 (+ 6001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.67:48889: BlockingRecv error: recv got EOF from 127.21.201.67:48889 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260812 00:59:29.779332 5876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:29.764646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.67:48889)
0812 00:59:29.765386 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:29.765435 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:29.767175 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:29.767925 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:29.767934 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:29.768481 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:29.769064 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:29.769081 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.770139 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.770143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:29.771449 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:29.771462 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.772441 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.772451 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:29.772784 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:29.773689 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:29.773722 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:29.778114 (+ 4392us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":302,"threads_started":1}
W20260812 00:59:29.780107 5874 heartbeater.cc:644] Failed to heartbeat to 127.21.201.126:38217 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.201.126:38217: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Aug 12 00:59:29 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:29 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:29.789009 5876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:29.780767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.67:41021)
0812 00:59:29.780950 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:29.780975 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:29.781080 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:29.781429 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:29.781435 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:29.781783 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:29.781929 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:29.781937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.782946 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.782951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:29.783903 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:29.783922 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.784673 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.784679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:29.784862 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:29.785374 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:29.785391 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:29.788692 (+ 3301us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 00:59:29.789206 5381 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:29.780991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:41021 (local address 127.21.201.126:38217)
0812 00:59:29.781181 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:29.781186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:29.781201 (+ 15us) server_negotiation.cc:407] Connection header received
0812 00:59:29.781249 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:29.781252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:29.781301 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:29.781401 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:29.782065 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.782798 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.784048 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.784523 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.784659 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:29.788934 (+ 4275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.67:41021: BlockingRecv error: recv got EOF from 127.21.201.67:41021 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 00:59:29.789798 5880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:29.772974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48424 (local address 127.21.201.126:38217)
0812 00:59:29.773604 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:29.773611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:29.774141 (+ 530us) server_negotiation.cc:407] Connection header received
0812 00:59:29.774269 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:29.774276 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:29.774352 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:29.774476 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:29.776190 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.777341 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.778668 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.779164 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.779305 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:29.782311 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:29.782320 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:29.782338 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:29.782373 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:29.785786 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:29.786687 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:29.786695 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:29.786703 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:29.786993 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:29.787402 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:29.787407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:29.787411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:29.788123 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:29.788447 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:29.788947 (+ 500us) server_negotiation.cc:299] Negotiation successful
0812 00:59:29.789375 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":327,"threads_started":1}
Aug 12 00:59:29 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:29 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:29.800818 5876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:29.790085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.67:40255)
0812 00:59:29.790235 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:29.790255 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:29.790358 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:29.791578 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:29.791587 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:29.792212 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:29.792444 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:29.792460 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.793711 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.793717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:29.794931 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:29.794945 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.796044 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.796048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:29.796209 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:29.796869 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:29.796887 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:29.800464 (+ 3577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260812 00:59:29.800979 5880 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:29.790940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:40255 (local address 127.21.201.126:38217)
0812 00:59:29.791171 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:29.791175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:29.791190 (+ 15us) server_negotiation.cc:407] Connection header received
0812 00:59:29.791249 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:29.791257 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:29.791308 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:29.791387 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:29.792634 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.793517 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.795164 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.795852 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.796010 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:29.800701 (+ 4691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.67:40255: BlockingRecv error: recv got EOF from 127.21.201.67:40255 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
W20260812 00:59:29.801183 5874 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 12 00:59:29 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:29 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:29.808671 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:29.798968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.65:60359)
0812 00:59:29.799559 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:29.799597 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:29.799782 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:29.800173 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:29.800176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:29.800726 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:29.801051 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:29.801064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.802138 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.802141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:29.803111 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:29.803121 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.803987 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.803995 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:29.804145 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:29.804774 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:29.804800 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:29.808414 (+ 3614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":367,"threads_started":1}
W20260812 00:59:29.808884 5381 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:29.799117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:60359 (local address 127.21.201.126:38217)
0812 00:59:29.799349 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:29.799356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:29.799706 (+ 350us) server_negotiation.cc:407] Connection header received
0812 00:59:29.799921 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:29.799928 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:29.800010 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:29.800117 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:29.801217 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.801977 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.803258 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:29.803699 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:29.803880 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:29.808640 (+ 4760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.65:60359: BlockingRecv error: recv got EOF from 127.21.201.65:60359 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260812 00:59:30.260143 5616 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.21.201.66:38357
Aug 12 00:59:30 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:30 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:30.308579 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:30.298746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.66:51585)
0812 00:59:30.299564 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:30.299586 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:30.299718 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:30.300094 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:30.300097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:30.300578 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:30.300790 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:30.300799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:30.302266 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:30.302271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:30.303258 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:30.303271 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:30.304192 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:30.304198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:30.304366 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:30.305077 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:30.305110 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:30.308209 (+ 3099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":521,"threads_started":1}
W20260812 00:59:30.308830 5381 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:30.298873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:51585 (local address 127.21.201.126:38217)
0812 00:59:30.299084 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:30.299089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:30.299689 (+ 600us) server_negotiation.cc:407] Connection header received
0812 00:59:30.299866 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:30.299870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:30.299939 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:30.300028 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:30.300943 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:30.302108 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:30.303419 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:30.304010 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:30.304146 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:30.308586 (+ 4440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.66:51585: BlockingRecv error: recv got EOF from 127.21.201.66:51585 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 12 00:59:30 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:30 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:30.811869 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:30.801901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.67:46185)
0812 00:59:30.802482 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:30.802498 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:30.802669 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:30.803145 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:30.803150 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:30.803721 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:30.804000 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:30.804010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:30.805097 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:30.805103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:30.806242 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:30.806255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:30.807088 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:30.807093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:30.807217 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:30.807897 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:30.807915 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:30.811481 (+ 3566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":310,"threads_started":1}
W20260812 00:59:30.812059 5381 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:30.802015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:46185 (local address 127.21.201.126:38217)
0812 00:59:30.802330 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:30.802337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:30.802590 (+ 253us) server_negotiation.cc:407] Connection header received
0812 00:59:30.802860 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:30.802868 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:30.802950 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:30.803055 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:30.804163 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:30.804880 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:30.806412 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:30.806917 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:30.807033 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:30.811765 (+ 4732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.67:46185: BlockingRecv error: recv got EOF from 127.21.201.67:46185 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 12 00:59:30 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:30 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:30.819296 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:30.809813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.65:48143)
0812 00:59:30.810626 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:30.810650 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:30.810799 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:30.811668 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:30.811675 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:30.812223 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:30.812470 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:30.812496 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:30.813556 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:30.813561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:30.814451 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:30.814462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:30.815216 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:30.815222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:30.815403 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:30.816079 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:30.816097 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:30.818909 (+ 2812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":529,"threads_started":1}
W20260812 00:59:30.819451 5892 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:30.809892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:48143 (local address 127.21.201.126:38217)
0812 00:59:30.811316 (+ 1424us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:30.811322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:30.811341 (+ 19us) server_negotiation.cc:407] Connection header received
0812 00:59:30.811390 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:30.811397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:30.811475 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:30.812223 (+ 748us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:30.812650 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:30.813391 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:30.814605 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:30.815043 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:30.815175 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:30.819157 (+ 3982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.65:48143: BlockingRecv error: recv got EOF from 127.21.201.65:48143 (error 108)
Metrics: {"server-negotiator.queue_time_us":1294,"thread_start_us":1122,"threads_started":1}
Aug 12 00:59:31 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:31 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:31.319837 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:31.309728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.66:42273)
0812 00:59:31.310395 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:31.310418 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:31.310545 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:31.310927 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:31.310931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:31.311415 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:31.311600 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:31.311608 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:31.312843 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:31.312848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:31.314293 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:31.314317 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:31.315273 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:31.315278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:31.315410 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:31.316161 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:31.316179 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:31.319399 (+ 3220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":396,"threads_started":1}
W20260812 00:59:31.319901 5892 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:31.309850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:42273 (local address 127.21.201.126:38217)
0812 00:59:31.310069 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:31.310075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:31.310515 (+ 440us) server_negotiation.cc:407] Connection header received
0812 00:59:31.310662 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:31.310666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:31.310735 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:31.310839 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:31.311805 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:31.312664 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:31.314460 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:31.315107 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:31.315210 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:31.319633 (+ 4423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.66:42273: BlockingRecv error: recv got EOF from 127.21.201.66:42273 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 12 00:59:31 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:31 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 00:59:31.822104 5892 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:31.813159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:54953 (local address 127.21.201.126:38217)
0812 00:59:31.813447 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:31.813454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:31.813675 (+ 221us) server_negotiation.cc:407] Connection header received
0812 00:59:31.813814 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:31.813818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:31.813904 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:31.813986 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:31.814772 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:31.815521 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:31.816762 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:31.817246 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:31.817349 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:31.821732 (+ 4383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.67:54953: BlockingRecv error: recv got EOF from 127.21.201.67:54953 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 00:59:31.822345 5896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:31.813006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.67:54953)
0812 00:59:31.813576 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:31.813594 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:31.813711 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:31.814044 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:31.814048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:31.814461 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:31.814646 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:31.814655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:31.815635 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:31.815639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:31.816632 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:31.816644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:31.817383 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:31.817387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:31.817501 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:31.818179 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:31.818197 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:31.821968 (+ 3771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":334,"threads_started":1}
Aug 12 00:59:31 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:31 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:31.829612 5897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:31.820470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.65:43879)
0812 00:59:31.821371 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:31.821393 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:31.821519 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:31.822396 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:31.822403 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:31.822843 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:31.823028 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:31.823038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:31.824106 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:31.824110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:31.825037 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:31.825049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:31.825695 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:31.825699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:31.825832 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:31.826479 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:31.826497 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:31.829341 (+ 2844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":757,"thread_start_us":615,"threads_started":1}
W20260812 00:59:31.829967 5898 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:31.820908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:43879 (local address 127.21.201.126:38217)
0812 00:59:31.822026 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:31.822030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:31.822045 (+ 15us) server_negotiation.cc:407] Connection header received
0812 00:59:31.822085 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:31.822088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:31.822159 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:31.822353 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:31.823166 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:31.823990 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:31.825167 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:31.825568 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:31.825659 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:31.829630 (+ 3971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.65:43879: BlockingRecv error: recv got EOF from 127.21.201.65:43879 (error 108)
Metrics: {"server-negotiator.queue_time_us":1029,"thread_start_us":894,"threads_started":1}
Aug 12 00:59:32 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:32 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 00:59:32.330292 5898 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:32.321054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:42335 (local address 127.21.201.126:38217)
0812 00:59:32.321252 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:32.321256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:32.321776 (+ 520us) server_negotiation.cc:407] Connection header received
0812 00:59:32.321997 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:32.322002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:32.322079 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:32.322187 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:32.323126 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:32.323966 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:32.325164 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:32.325593 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:32.325728 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:32.330088 (+ 4360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.66:42335: BlockingRecv error: recv got EOF from 127.21.201.66:42335 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 00:59:32.330303 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:32.320970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.66:42335)
0812 00:59:32.321668 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:32.321690 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:32.321848 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:32.322220 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:32.322224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:32.322709 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:32.322968 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:32.322979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:32.324098 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:32.324103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:32.325016 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:32.325028 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:32.325746 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:32.325751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:32.325944 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:32.326625 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:32.326642 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:32.329907 (+ 3265us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":412,"threads_started":1}
Aug 12 00:59:32 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:32 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:32.833444 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:32.823643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.67:56287)
0812 00:59:32.824334 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:32.824351 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:32.824523 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:32.824892 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:32.824898 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:32.825304 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:32.825486 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:32.825494 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:32.826603 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:32.826610 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:32.827539 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:32.827551 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:32.828375 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:32.828381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:32.828563 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:32.829270 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:32.829288 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:32.833037 (+ 3749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":448,"threads_started":1}
W20260812 00:59:32.833544 5898 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:32.823815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:56287 (local address 127.21.201.126:38217)
0812 00:59:32.824181 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:32.824188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:32.824449 (+ 261us) server_negotiation.cc:407] Connection header received
0812 00:59:32.824667 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:32.824671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:32.824735 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:32.824847 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:32.825643 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:32.826460 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:32.827693 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:32.828217 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:32.828349 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:32.833266 (+ 4917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.67:56287: BlockingRecv error: recv got EOF from 127.21.201.67:56287 (error 108)
Metrics: {"server-negotiator.queue_time_us":181}
Aug 12 00:59:32 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:32 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:32.840559 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:32.830816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.65:32835)
0812 00:59:32.831559 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:32.831583 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:32.831722 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:32.832265 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:32.832268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:32.832709 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:32.832862 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:32.832870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:32.834218 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:32.834223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:32.835209 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:32.835220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:32.835987 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:32.835993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:32.836187 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:32.836837 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:32.836861 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:32.840204 (+ 3343us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":447,"threads_started":1}
W20260812 00:59:32.840750 5912 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:32.831205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:32835 (local address 127.21.201.126:38217)
0812 00:59:32.831882 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:32.831895 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:32.831909 (+ 14us) server_negotiation.cc:407] Connection header received
0812 00:59:32.831979 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:32.831984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:32.832065 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:32.832186 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:32.833034 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:32.834080 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:32.835359 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:32.835836 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:32.835966 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:32.840424 (+ 4458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.65:32835: BlockingRecv error: recv got EOF from 127.21.201.65:32835 (error 108)
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":364,"threads_started":1}
Aug 12 00:59:33 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:33 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:33.340605 5917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:33.331594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.66:47401)
0812 00:59:33.332203 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:33.332223 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:33.332398 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:33.332809 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:33.332813 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:33.333221 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:33.333379 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:33.333387 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:33.334566 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:33.334573 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:33.335593 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:33.335606 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:33.336374 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:33.336379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:33.336514 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:33.337195 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:33.337215 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:33.340354 (+ 3139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":330,"threads_started":1}
W20260812 00:59:33.340822 5912 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:33.331662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:47401 (local address 127.21.201.126:38217)
0812 00:59:33.331913 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:33.331918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:33.332343 (+ 425us) server_negotiation.cc:407] Connection header received
0812 00:59:33.332555 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:33.332561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:33.332646 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:33.332757 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:33.333537 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:33.334412 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:33.335773 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:33.336188 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:33.336295 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:33.340592 (+ 4297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.66:47401: BlockingRecv error: recv got EOF from 127.21.201.66:47401 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Aug 12 00:59:33 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:33 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:33.845876 5922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:33.834703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.67:59181)
0812 00:59:33.835468 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:33.835502 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:33.835817 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:33.836310 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:33.836316 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:33.837013 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:33.837352 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:33.837378 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:33.838690 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:33.838697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:33.839717 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:33.839728 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:33.840527 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:33.840533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:33.840711 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:33.841570 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:33.841596 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:33.845549 (+ 3953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":326,"threads_started":1}
W20260812 00:59:33.846143 5912 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:33.834807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:59181 (local address 127.21.201.126:38217)
0812 00:59:33.835105 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:33.835111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:33.835716 (+ 605us) server_negotiation.cc:407] Connection header received
0812 00:59:33.836022 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:33.836027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:33.836135 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:33.836273 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:33.837602 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:33.838522 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:33.839929 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:33.840377 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:33.840515 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:33.845826 (+ 5311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.67:59181: BlockingRecv error: recv got EOF from 127.21.201.67:59181 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Aug 12 00:59:33 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:33 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:33.852311 5924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:33.841806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.65:44325)
0812 00:59:33.843064 (+ 1258us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:33.843089 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:33.843237 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:33.843568 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:33.843574 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:33.844145 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:33.844325 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:33.844335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:33.845721 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:33.845728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:33.847010 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:33.847024 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:33.847783 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:33.847797 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:33.847927 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:33.848605 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:33.848624 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:33.852049 (+ 3425us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1078,"thread_start_us":868,"threads_started":1}
W20260812 00:59:33.852607 5925 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:33.841898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:44325 (local address 127.21.201.126:38217)
0812 00:59:33.843272 (+ 1374us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:33.843276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:33.843295 (+ 19us) server_negotiation.cc:407] Connection header received
0812 00:59:33.843344 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:33.843347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:33.843410 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:33.843499 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:33.844495 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:33.845557 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:33.847168 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:33.847601 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:33.847704 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:33.852284 (+ 4580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.65:44325: BlockingRecv error: recv got EOF from 127.21.201.65:44325 (error 108)
Metrics: {"server-negotiator.queue_time_us":1241,"thread_start_us":1068,"threads_started":1}
Aug 12 00:59:34 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:34 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:34.351259 5926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:34.341762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.66:54587)
0812 00:59:34.342384 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:34.342413 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:34.342570 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:34.342965 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:34.342970 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:34.343538 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:34.343822 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:34.343836 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:34.345065 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:34.345073 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:34.345995 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:34.346005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:34.346962 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:34.346967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:34.347097 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:34.347898 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:34.347918 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:34.350858 (+ 2940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":307,"threads_started":1}
W20260812 00:59:34.351318 5925 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:34.342086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:54587 (local address 127.21.201.126:38217)
0812 00:59:34.342273 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:34.342281 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:34.342518 (+ 237us) server_negotiation.cc:407] Connection header received
0812 00:59:34.342707 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:34.342714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:34.342801 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:34.342902 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:34.344007 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:34.344911 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:34.346148 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:34.346751 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:34.346882 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:34.351087 (+ 4205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.66:54587: BlockingRecv error: recv got EOF from 127.21.201.66:54587 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 12 00:59:34 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:34 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:34.861011 5929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:34.847113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.67:51695)
0812 00:59:34.847819 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:34.847837 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:34.847967 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:34.848348 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:34.848352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:34.848809 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:34.848993 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:34.849002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:34.850051 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:34.850056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:34.850993 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:34.851004 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:34.851827 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:34.851837 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:34.852083 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:34.857087 (+ 5004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:34.857106 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:34.860638 (+ 3532us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":356,"threads_started":1}
Aug 12 00:59:34 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 00:59:34.861107 5925 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:34.847236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:51695 (local address 127.21.201.126:38217)
0812 00:59:34.847474 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:34.847479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:34.847954 (+ 475us) server_negotiation.cc:407] Connection header received
0812 00:59:34.848107 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:34.848115 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:34.848206 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:34.848290 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:34.849163 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:34.849914 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:34.851148 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:34.851614 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:34.851711 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:34.860843 (+ 9132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.67:51695: BlockingRecv error: recv got EOF from 127.21.201.67:51695 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Aug 12 00:59:34 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:34.863207 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:34.853346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.65:41467)
0812 00:59:34.853980 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:34.854004 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:34.854136 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:34.854777 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:34.854785 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:34.855266 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:34.855422 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:34.855432 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:34.856684 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:34.856691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:34.857764 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:34.857778 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:34.858922 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:34.858927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:34.859205 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:34.859838 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:34.859868 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:34.862873 (+ 3005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":345,"threads_started":1}
W20260812 00:59:34.863498 5931 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:34.853455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:41467 (local address 127.21.201.126:38217)
0812 00:59:34.854475 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:34.854479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:34.854494 (+ 15us) server_negotiation.cc:407] Connection header received
0812 00:59:34.854538 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:34.854541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:34.854612 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:34.854737 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:34.855589 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:34.856521 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:34.857932 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:34.858680 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:34.858821 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:34.863090 (+ 4269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.65:41467: BlockingRecv error: recv got EOF from 127.21.201.65:41467 (error 108)
Metrics: {"server-negotiator.queue_time_us":926,"thread_start_us":789,"threads_started":1}
Aug 12 00:59:35 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:35 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:35.361495 5932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:35.352435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.66:42547)
0812 00:59:35.353059 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:35.353081 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:35.353257 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:35.353629 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:35.353637 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:35.354101 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:35.354263 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:35.354272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:35.355340 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:35.355346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:35.356327 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:35.356339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:35.357263 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:35.357269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:35.357449 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:35.358153 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:35.358171 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:35.361160 (+ 2989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":342,"threads_started":1}
W20260812 00:59:35.361733 5931 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:35.352581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:42547 (local address 127.21.201.126:38217)
0812 00:59:35.352833 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:35.352842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:35.353178 (+ 336us) server_negotiation.cc:407] Connection header received
0812 00:59:35.353396 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:35.353402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:35.353475 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:35.353582 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:35.354417 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:35.355187 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:35.356481 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:35.357119 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:35.357234 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:35.361399 (+ 4165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.66:42547: BlockingRecv error: recv got EOF from 127.21.201.66:42547 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 12 00:59:35 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:35 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:35 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:35.874819 5936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:35.862655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.67:55353)
0812 00:59:35.863394 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:35.863415 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:35.863577 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:35.864066 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:35.864071 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:35.864625 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:35.864791 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:35.864801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:35.866262 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:35.866267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:35.867548 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:35.867567 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:35.868627 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:35.868632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:35.868774 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:35.869531 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:35.869556 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:35.874572 (+ 5016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":493,"threads_started":1}
Aug 12 00:59:35 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 00:59:35.875883 5935 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:35.862688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:55353 (local address 127.21.201.126:38217)
0812 00:59:35.863313 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:35.863319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:35.863516 (+ 197us) server_negotiation.cc:407] Connection header received
0812 00:59:35.863708 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:35.863713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:35.863850 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:35.864031 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:35.864977 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:35.866078 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:35.867814 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:35.868426 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:35.868566 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:35.874904 (+ 6338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.67:55353: BlockingRecv error: recv got EOF from 127.21.201.67:55353 (error 108)
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":302,"threads_started":1}
I20260812 00:59:35.876426 5937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:35.864314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.65:32823)
0812 00:59:35.865391 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:35.865415 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:35.865550 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:35.866023 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:35.866028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:35.866529 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:35.866706 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:35.866716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:35.868242 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:35.868250 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:35.869390 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:35.869407 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:35.870898 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:35.870904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:35.871244 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:35.871956 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:35.871987 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:35.876116 (+ 4129us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":924,"thread_start_us":704,"threads_started":1}
W20260812 00:59:35.876788 5939 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:35.864415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:32823 (local address 127.21.201.126:38217)
0812 00:59:35.865728 (+ 1313us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:35.865733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:35.865749 (+ 16us) server_negotiation.cc:407] Connection header received
0812 00:59:35.865790 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:35.865793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:35.865851 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:35.865944 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:35.866902 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:35.868074 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:35.869548 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:35.870720 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:35.870863 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:35.876428 (+ 5565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.65:32823: BlockingRecv error: recv got EOF from 127.21.201.65:32823 (error 108)
Metrics: {"server-negotiator.queue_time_us":1225,"thread_start_us":1091,"threads_started":1}
Aug 12 00:59:36 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:36 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:36.372009 5940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:36.362957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.66:41109)
0812 00:59:36.363618 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:36.363638 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:36.363815 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:36.364184 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:36.364189 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:36.364651 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:36.364825 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:36.364833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:36.365881 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:36.365915 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:36.366871 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:36.366883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:36.367617 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:36.367624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:36.367833 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:36.368504 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:36.368523 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:36.371583 (+ 3060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":314,"threads_started":1}
W20260812 00:59:36.372135 5939 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:36.362979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:41109 (local address 127.21.201.126:38217)
0812 00:59:36.363207 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:36.363213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:36.363702 (+ 489us) server_negotiation.cc:407] Connection header received
0812 00:59:36.363953 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:36.363960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:36.364042 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:36.364151 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:36.364972 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:36.365726 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:36.367016 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:36.367445 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:36.367578 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:36.371860 (+ 4282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.66:41109: BlockingRecv error: recv got EOF from 127.21.201.66:41109 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 12 00:59:36 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:36 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:36 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:36 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:36.888155 5942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:36.876164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.67:33683)
0812 00:59:36.876897 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:36.876922 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:36.877071 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:36.877648 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:36.877654 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:36.878724 (+ 1070us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:36.879009 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:36.879025 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:36.880756 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:36.880763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:36.882225 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:36.882238 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:36.883208 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:36.883215 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:36.883401 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:36.884224 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:36.884243 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:36.887637 (+ 3394us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":299,"threads_started":1}
W20260812 00:59:36.888293 5944 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:36.876487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:33683 (local address 127.21.201.126:38217)
0812 00:59:36.877234 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:36.877242 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:36.877262 (+ 20us) server_negotiation.cc:407] Connection header received
0812 00:59:36.877332 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:36.877340 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:36.877459 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:36.877577 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:36.879265 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:36.880560 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:36.882366 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:36.883041 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:36.883137 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:36.887965 (+ 4828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.67:33683: BlockingRecv error: recv got EOF from 127.21.201.67:33683 (error 108)
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":492,"threads_started":1}
I20260812 00:59:36.888657 5945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:36.877505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.65:54867)
0812 00:59:36.878171 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:36.878191 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:36.878307 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:36.879027 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:36.879032 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:36.879594 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:36.879801 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:36.879812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:36.881455 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:36.881464 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:36.882777 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:36.882792 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:36.883908 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:36.883915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:36.884102 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:36.884650 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:36.884668 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:36.888314 (+ 3646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":813,"threads_started":1}
W20260812 00:59:36.889380 5947 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:36.877703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:54867 (local address 127.21.201.126:38217)
0812 00:59:36.878763 (+ 1060us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:36.878767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:36.878781 (+ 14us) server_negotiation.cc:407] Connection header received
0812 00:59:36.878818 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:36.878821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:36.878866 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:36.878988 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:36.880027 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:36.881265 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:36.882972 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:36.883681 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:36.883850 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:36.889171 (+ 5321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.65:54867: BlockingRecv error: recv got EOF from 127.21.201.65:54867 (error 108)
Metrics: {"server-negotiator.queue_time_us":966,"thread_start_us":872,"threads_started":1}
Aug 12 00:59:37 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:37 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:37.382570 5948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:37.373194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.66:36733)
0812 00:59:37.373846 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:37.373867 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:37.374001 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:37.374308 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:37.374311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:37.374736 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:37.374911 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:37.374920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:37.376246 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:37.376251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:37.377218 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:37.377231 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:37.377965 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:37.377970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:37.378086 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:37.378802 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:37.378820 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:37.382205 (+ 3385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":376,"threads_started":1}
W20260812 00:59:37.382656 5947 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:37.373246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:36733 (local address 127.21.201.126:38217)
0812 00:59:37.373460 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:37.373465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:37.373943 (+ 478us) server_negotiation.cc:407] Connection header received
0812 00:59:37.374111 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:37.374114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:37.374177 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:37.374260 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:37.375089 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:37.376080 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:37.377371 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:37.377786 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:37.377895 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:37.382419 (+ 4524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.66:36733: BlockingRecv error: recv got EOF from 127.21.201.66:36733 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 12 00:59:37 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:37 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:37 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 00:59:37.913736 5956 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:37.890751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:58077 (local address 127.21.201.126:38217)
0812 00:59:37.900301 (+ 9550us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:37.900307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:37.902638 (+ 2331us) server_negotiation.cc:407] Connection header received
0812 00:59:37.902688 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:37.902691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:37.902771 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:37.902887 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:37.904568 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:37.905401 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:37.907315 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:37.907725 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:37.907905 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:37.913377 (+ 5472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.67:58077: BlockingRecv error: recv got EOF from 127.21.201.67:58077 (error 108)
Metrics: {"server-negotiator.queue_time_us":9418,"thread_start_us":13173,"threads_started":1}
Aug 12 00:59:37 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:37.914306 5949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:37.889388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.67:58077)
0812 00:59:37.902347 (+ 12959us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:37.902382 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:37.902534 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:37.902921 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:37.902927 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:37.903561 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:37.903829 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:37.903846 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:37.905600 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:37.905606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:37.907184 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:37.907201 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:37.907971 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:37.907980 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:37.908150 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:37.909052 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:37.909097 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:37.913983 (+ 4886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":12744,"thread_start_us":13676,"threads_started":1}
I20260812 00:59:37.915272 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:37.891265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.65:38455)
0812 00:59:37.893341 (+ 2076us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:37.893379 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:37.893552 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:37.904552 (+ 11000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:37.904561 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:37.905251 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:37.905594 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:37.905615 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:37.907142 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:37.907150 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:37.909074 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:37.909097 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:37.910156 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:37.910164 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:37.910396 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:37.911046 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:37.911085 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:37.914883 (+ 3798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1900,"thread_start_us":1715,"threads_started":1}
W20260812 00:59:37.916253 5952 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:37.891300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:38455 (local address 127.21.201.126:38217)
0812 00:59:37.904173 (+ 12873us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:37.904177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:37.904191 (+ 14us) server_negotiation.cc:407] Connection header received
0812 00:59:37.904227 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:37.904230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:37.904276 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:37.904438 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:37.905763 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:37.906967 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:37.909258 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:37.909915 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:37.910104 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:37.915844 (+ 5740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.65:38455: BlockingRecv error: recv got EOF from 127.21.201.65:38455 (error 108)
Metrics: {"server-negotiator.queue_time_us":12769,"thread_start_us":8744,"threads_started":1}
Aug 12 00:59:38 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:38 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:38.392894 5957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:38.383678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.66:54337)
0812 00:59:38.384236 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:38.384257 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:38.384381 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:38.384676 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:38.384679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:38.385110 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:38.385288 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:38.385297 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:38.386409 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:38.386413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:38.387481 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:38.387493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:38.388268 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:38.388273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:38.388465 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:38.389162 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:38.389181 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:38.392629 (+ 3448us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":316,"threads_started":1}
W20260812 00:59:38.393175 5952 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:38.383729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:54337 (local address 127.21.201.126:38217)
0812 00:59:38.383990 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:38.383999 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:38.384338 (+ 339us) server_negotiation.cc:407] Connection header received
0812 00:59:38.384482 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:38.384485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:38.384549 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:38.384632 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:38.385447 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:38.386279 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:38.387617 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:38.388110 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:38.388209 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:38.392863 (+ 4654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.66:54337: BlockingRecv error: recv got EOF from 127.21.201.66:54337 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 12 00:59:38 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:38 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:38 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:38 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:38.926532 5959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:38.915992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.67:41653)
0812 00:59:38.916700 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:38.916722 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:38.916870 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:38.917574 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:38.917579 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:38.918044 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:38.918218 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:38.918227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:38.919392 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:38.919400 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:38.920801 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:38.920817 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:38.922046 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:38.922052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:38.922194 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:38.922875 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:38.922894 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:38.926233 (+ 3339us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":427,"threads_started":1}
I20260812 00:59:38.926877 5961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:38.916629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.65:39635)
0812 00:59:38.917192 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:38.917212 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:38.917333 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:38.917688 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:38.917696 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:38.918103 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:38.918253 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:38.918262 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:38.919517 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:38.919523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:38.920863 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:38.920895 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:38.921945 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:38.921954 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:38.922128 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:38.922969 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:38.922987 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:38.926550 (+ 3563us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":327,"threads_started":1}
W20260812 00:59:38.927102 5958 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:38.915870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:41653 (local address 127.21.201.126:38217)
0812 00:59:38.916548 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:38.916556 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:38.917177 (+ 621us) server_negotiation.cc:407] Connection header received
0812 00:59:38.917265 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:38.917269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:38.917399 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:38.917538 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:38.918374 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:38.919240 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:38.921128 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:38.921870 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:38.922007 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:38.926490 (+ 4483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.67:41653: BlockingRecv error: recv got EOF from 127.21.201.67:41653 (error 108)
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":429,"thread_start_us":296,"threads_started":1}
W20260812 00:59:38.927805 5962 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:38.916747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:39635 (local address 127.21.201.126:38217)
0812 00:59:38.917401 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:38.917405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:38.917419 (+ 14us) server_negotiation.cc:407] Connection header received
0812 00:59:38.917460 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:38.917463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:38.917515 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:38.917636 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:38.918420 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:38.919374 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:38.921028 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:38.921778 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:38.921902 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:38.927159 (+ 5257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.65:39635: BlockingRecv error: recv got EOF from 127.21.201.65:39635 (error 108)
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":563,"thread_start_us":421,"threads_started":1}
Aug 12 00:59:39 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:39 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:39.402820 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:39.394003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.66:48343)
0812 00:59:39.394583 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:39.394615 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:39.394782 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:39.395131 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:39.395136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:39.395598 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:39.395775 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:39.395783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:39.396824 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:39.396830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:39.397889 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:39.397902 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:39.398646 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:39.398657 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:39.398835 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:39.399491 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:39.399509 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:39.402474 (+ 2965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":323,"threads_started":1}
W20260812 00:59:39.402925 5962 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:39.394088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:48343 (local address 127.21.201.126:38217)
0812 00:59:39.394383 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:39.394393 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:39.394707 (+ 314us) server_negotiation.cc:407] Connection header received
0812 00:59:39.394911 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:39.394916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:39.394992 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:39.395100 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:39.395925 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:39.396678 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:39.398046 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:39.398447 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:39.398585 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:39.402680 (+ 4095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.66:48343: BlockingRecv error: recv got EOF from 127.21.201.66:48343 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Aug 12 00:59:39 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:39 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:39 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:39 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:39.941685 5965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:39.927812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.67:44019)
0812 00:59:39.929159 (+ 1347us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:39.929183 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:39.929346 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:39.930140 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:39.930144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:39.930517 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:39.930663 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:39.930672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:39.931875 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:39.931882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:39.932971 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:39.932984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:39.933962 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:39.933969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:39.934147 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:39.937832 (+ 3685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:39.937858 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:39.941434 (+ 3576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1213,"thread_start_us":1104,"threads_started":1}
W20260812 00:59:39.942081 5967 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:39.928253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:44019 (local address 127.21.201.126:38217)
0812 00:59:39.929781 (+ 1528us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:39.929785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:39.929800 (+ 15us) server_negotiation.cc:407] Connection header received
0812 00:59:39.929894 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:39.929897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:39.929966 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:39.930087 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:39.930819 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:39.931672 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:39.933134 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:39.933777 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:39.933926 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:39.941715 (+ 7789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.67:44019: BlockingRecv error: recv got EOF from 127.21.201.67:44019 (error 108)
Metrics: {"server-negotiator.queue_time_us":1431,"spinlock_wait_cycles":1280,"thread_start_us":1074,"threads_started":1}
I20260812 00:59:39.942170 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:39.927840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.65:47363)
0812 00:59:39.928877 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:39.928905 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:39.929072 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:39.929964 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:39.929969 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:39.930514 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:39.930706 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:39.930716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:39.932231 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:39.932238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:39.933142 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:39.933154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:39.934243 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:39.934250 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:39.934507 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:39.937832 (+ 3325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:39.937858 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:39.941829 (+ 3971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":861,"thread_start_us":692,"threads_started":1}
W20260812 00:59:39.942682 5968 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:39.928030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:47363 (local address 127.21.201.126:38217)
0812 00:59:39.929628 (+ 1598us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:39.929634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:39.929653 (+ 19us) server_negotiation.cc:407] Connection header received
0812 00:59:39.929727 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:39.929730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:39.929792 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:39.929886 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:39.930891 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:39.932087 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:39.933297 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:39.934075 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:39.934212 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:39.942389 (+ 8177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.65:47363: BlockingRecv error: recv got EOF from 127.21.201.65:47363 (error 108)
Metrics: {"server-negotiator.queue_time_us":1444,"thread_start_us":1483,"threads_started":1}
Aug 12 00:59:40 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:40 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:40.413201 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:40.404100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.66:51535)
0812 00:59:40.404788 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:40.404817 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:40.404978 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:40.405360 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:40.405365 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:40.405833 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:40.406034 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:40.406043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:40.407153 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:40.407161 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:40.408134 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:40.408146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:40.408895 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:40.408901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:40.409103 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:40.409748 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:40.409767 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:40.412914 (+ 3147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":380,"threads_started":1}
W20260812 00:59:40.413650 5968 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:40.404144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:51535 (local address 127.21.201.126:38217)
0812 00:59:40.404378 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:40.404385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:40.404903 (+ 518us) server_negotiation.cc:407] Connection header received
0812 00:59:40.405115 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:40.405120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:40.405227 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:40.405329 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:40.406186 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:40.407011 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:40.408300 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:40.408720 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:40.408867 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:40.413421 (+ 4554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.66:51535: BlockingRecv error: recv got EOF from 127.21.201.66:51535 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 12 00:59:40 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:40 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:40 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:40 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:40.953902 5972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:40.942843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.67:50077)
0812 00:59:40.943536 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:40.943565 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:40.943769 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:40.944845 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:40.944851 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:40.945344 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:40.945539 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:40.945548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:40.946889 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:40.946893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:40.947853 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:40.947884 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:40.948766 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:40.948771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:40.948903 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:40.949696 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:40.949713 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:40.953551 (+ 3838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":327,"threads_started":1}
W20260812 00:59:40.954136 5980 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:40.942896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:50077 (local address 127.21.201.126:38217)
0812 00:59:40.944532 (+ 1636us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:40.944536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:40.944551 (+ 15us) server_negotiation.cc:407] Connection header received
0812 00:59:40.944602 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:40.944605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:40.944665 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:40.944754 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:40.945711 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:40.946724 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:40.948020 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:40.948594 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:40.948716 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:40.953829 (+ 5113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.67:50077: BlockingRecv error: recv got EOF from 127.21.201.67:50077 (error 108)
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":1499,"thread_start_us":1467,"threads_started":1}
I20260812 00:59:40.954301 5979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:40.943220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.65:34145)
0812 00:59:40.943728 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:40.943771 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:40.943901 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:40.944927 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:40.944932 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:40.945377 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:40.945537 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:40.945546 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:40.946720 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:40.946726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:40.947730 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:40.947801 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:40.948635 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:40.948644 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:40.948852 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:40.949625 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:40.949642 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:40.953966 (+ 4324us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":452,"threads_started":1}
W20260812 00:59:40.954872 5976 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:40.943568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:34145 (local address 127.21.201.126:38217)
0812 00:59:40.944595 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:40.944600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:40.944615 (+ 15us) server_negotiation.cc:407] Connection header received
0812 00:59:40.944657 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:40.944659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:40.944709 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:40.944840 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:40.945704 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:40.946563 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:40.947945 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:40.948476 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:40.948591 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:40.954685 (+ 6094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.65:34145: BlockingRecv error: recv got EOF from 127.21.201.65:34145 (error 108)
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":935,"thread_start_us":559,"threads_started":1}
Aug 12 00:59:41 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:41 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:41.425657 5981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:41.414333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.66:45791)
0812 00:59:41.414990 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:41.415038 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:41.415229 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:41.415628 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:41.415632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:41.416522 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:41.416696 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:41.416705 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:41.417828 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:41.417836 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:41.419211 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:41.419227 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:41.420534 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:41.420539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:41.420679 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:41.421505 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:41.421524 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:41.425394 (+ 3870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":434,"threads_started":1}
W20260812 00:59:41.425906 5976 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:41.414447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:45791 (local address 127.21.201.126:38217)
0812 00:59:41.414721 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:41.414729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:41.415144 (+ 415us) server_negotiation.cc:407] Connection header received
0812 00:59:41.415378 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:41.415384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:41.415466 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:41.415579 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:41.416880 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:41.417676 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:41.419516 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:41.420348 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:41.420478 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:41.425657 (+ 5179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.66:45791: BlockingRecv error: recv got EOF from 127.21.201.66:45791 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 12 00:59:41 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:41 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:41 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:41 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:41.966024 5982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:41.955256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.67:58745)
0812 00:59:41.955869 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:41.955892 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:41.956033 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:41.957299 (+ 1266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:41.957307 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:41.957901 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:41.958127 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:41.958142 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:41.959453 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:41.959460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:41.960458 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:41.960471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:41.961357 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:41.961366 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:41.961525 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:41.962358 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:41.962375 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:41.965662 (+ 3287us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":353,"threads_started":1}
W20260812 00:59:41.966223 5987 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:41.955391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:58745 (local address 127.21.201.126:38217)
0812 00:59:41.956985 (+ 1594us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:41.956991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:41.957006 (+ 15us) server_negotiation.cc:407] Connection header received
0812 00:59:41.957049 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:41.957056 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:41.957113 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:41.957949 (+ 836us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:41.958393 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:41.959275 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:41.960670 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:41.961188 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:41.961319 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:41.965908 (+ 4589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.67:58745: BlockingRecv error: recv got EOF from 127.21.201.67:58745 (error 108)
Metrics: {"mutex_wait_us":186,"server-negotiator.queue_time_us":1459,"thread_start_us":1120,"threads_started":1}
I20260812 00:59:41.966248 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:41.955310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.65:57419)
0812 00:59:41.956591 (+ 1281us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:41.956611 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:41.956749 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:41.957355 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:41.957364 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:41.957889 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:41.958144 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:41.958161 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:41.959421 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:41.959429 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:41.960412 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:41.960424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:41.961303 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:41.961311 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:41.961509 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:41.962360 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:41.962379 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:41.965893 (+ 3514us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1138,"thread_start_us":967,"threads_started":1}
W20260812 00:59:41.966703 5986 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:41.955389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:57419 (local address 127.21.201.126:38217)
0812 00:59:41.956610 (+ 1221us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:41.956617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:41.956794 (+ 177us) server_negotiation.cc:407] Connection header received
0812 00:59:41.956892 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:41.956898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:41.956990 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:41.957111 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:41.958339 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:41.959268 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:41.960570 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:41.961103 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:41.961196 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:41.966283 (+ 5087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.65:57419: BlockingRecv error: recv got EOF from 127.21.201.65:57419 (error 108)
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":1098,"thread_start_us":903,"threads_started":1}
Aug 12 00:59:42 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:42 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:42.436200 5989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:42.426908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.66:59539)
0812 00:59:42.427602 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:42.427631 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:42.427822 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:42.428191 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:42.428197 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:42.428668 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:42.428840 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:42.428849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:42.429930 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:42.429939 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:42.430909 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:42.430923 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:42.431701 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:42.431707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:42.431930 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:42.432634 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:42.432656 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:42.435907 (+ 3251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":549,"spinlock_wait_cycles":31104,"thread_start_us":469,"threads_started":1}
W20260812 00:59:42.436398 5986 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:42.427061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:59539 (local address 127.21.201.126:38217)
0812 00:59:42.427291 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:42.427297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:42.427730 (+ 433us) server_negotiation.cc:407] Connection header received
0812 00:59:42.427968 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:42.427974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:42.428048 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:42.428155 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:42.428991 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:42.429767 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:42.431074 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:42.431500 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:42.431648 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:42.436141 (+ 4493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.66:59539: BlockingRecv error: recv got EOF from 127.21.201.66:59539 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 12 00:59:42 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:42 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:42 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:42 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:42.982618 5996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:42.967376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.65:33329)
0812 00:59:42.971649 (+ 4273us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:42.971670 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:42.971849 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:42.972433 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:42.972441 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:42.972910 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:42.973198 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:42.973212 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:42.974745 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:42.974750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:42.976240 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:42.976256 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:42.977435 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:42.977443 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:42.977637 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:42.978612 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:42.978637 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:42.982223 (+ 3586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4148,"thread_start_us":4018,"threads_started":1}
I20260812 00:59:42.982820 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:42.967307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.67:34359)
0812 00:59:42.971408 (+ 4101us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:42.971434 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:42.971598 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:42.972160 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:42.972167 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:42.972727 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:42.972967 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:42.972979 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:42.974593 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:42.974597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:42.976239 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:42.976256 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:42.977475 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:42.977482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:42.977654 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:42.978590 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:42.978614 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:42.982467 (+ 3853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3907,"thread_start_us":3714,"threads_started":1}
W20260812 00:59:42.982931 5997 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:42.967389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:34359 (local address 127.21.201.126:38217)
0812 00:59:42.971087 (+ 3698us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:42.971094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:42.971719 (+ 625us) server_negotiation.cc:407] Connection header received
0812 00:59:42.971815 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:42.971819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:42.971914 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:42.972095 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:42.973130 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:42.974471 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:42.976414 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:42.977243 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:42.977411 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:42.982652 (+ 5241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.67:34359: BlockingRecv error: recv got EOF from 127.21.201.67:34359 (error 108)
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":3523,"spinlock_wait_cycles":3712,"thread_start_us":3343,"threads_started":1}
W20260812 00:59:42.983196 5998 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:42.967389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:33329 (local address 127.21.201.126:38217)
0812 00:59:42.971626 (+ 4237us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:42.971631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:42.971793 (+ 162us) server_negotiation.cc:407] Connection header received
0812 00:59:42.971989 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:42.971997 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:42.972068 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:42.972174 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:42.973329 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:42.974585 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:42.976414 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:42.977246 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:42.977410 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:42.982535 (+ 5125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.65:33329: BlockingRecv error: recv got EOF from 127.21.201.65:33329 (error 108)
Metrics: {"mutex_wait_us":179,"server-negotiator.queue_time_us":4134,"spinlock_wait_cycles":6656,"thread_start_us":4066,"threads_started":1}
Aug 12 00:59:43 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:43 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:43.445842 6004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:43.437279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.66:44495)
0812 00:59:43.437877 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:43.437902 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:43.438034 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:43.438405 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:43.438409 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:43.438888 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:43.439090 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:43.439100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:43.440247 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:43.440251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:43.441186 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:43.441197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:43.441930 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:43.441935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:43.442069 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:43.442736 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:43.442755 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:43.445579 (+ 2824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":340,"threads_started":1}
W20260812 00:59:43.445986 5998 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:43.437362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:44495 (local address 127.21.201.126:38217)
0812 00:59:43.437577 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:43.437586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:43.437994 (+ 408us) server_negotiation.cc:407] Connection header received
0812 00:59:43.438170 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:43.438174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:43.438242 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:43.438346 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:43.439255 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:43.440098 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:43.441341 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:43.441763 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:43.441861 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:43.445782 (+ 3921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.66:44495: BlockingRecv error: recv got EOF from 127.21.201.66:44495 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 12 00:59:43 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:43 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:43 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.67@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:43 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.65@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:43.997305 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:43.984225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.67:52747)
0812 00:59:43.985539 (+ 1314us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:43.985567 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:43.985760 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:43.987199 (+ 1439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:43.987205 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:43.988026 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:43.988317 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:43.988350 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:43.989827 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:43.989833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:43.990876 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:43.990890 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:43.991878 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:43.991885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:43.992157 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:43.992945 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:43.992987 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:43.996811 (+ 3824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1134,"thread_start_us":912,"threads_started":1}
I20260812 00:59:43.997674 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:43.984401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.65:37963)
0812 00:59:43.986012 (+ 1611us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:43.986032 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:43.986223 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:43.987330 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:43.987335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:43.988133 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:43.988394 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:43.988407 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:43.989955 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:43.989967 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:43.991014 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:43.991028 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:43.991998 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:43.992004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:43.992325 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:43.993185 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:43.993203 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:43.997048 (+ 3845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1396,"thread_start_us":1250,"threads_started":1}
W20260812 00:59:43.997674 6017 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:43.984429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:52747 (local address 127.21.201.126:38217)
0812 00:59:43.986687 (+ 2258us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:43.986694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:43.986713 (+ 19us) server_negotiation.cc:407] Connection header received
0812 00:59:43.986834 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:43.986846 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:43.986978 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:43.987174 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:43.988578 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:43.989562 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:43.991055 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:43.991653 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:43.991860 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:43.997130 (+ 5270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.67:52747: BlockingRecv error: recv got EOF from 127.21.201.67:52747 (error 108)
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":1892,"thread_start_us":1741,"threads_started":1}
W20260812 00:59:43.997932 6015 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:43.984471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:37963 (local address 127.21.201.126:38217)
0812 00:59:43.987004 (+ 2533us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:43.987012 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:43.987031 (+ 19us) server_negotiation.cc:407] Connection header received
0812 00:59:43.987082 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:43.987085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:43.987149 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:43.988062 (+ 913us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:43.988644 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:43.989761 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:43.991193 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:43.991825 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:43.991979 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:43.997493 (+ 5514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.65:37963: BlockingRecv error: recv got EOF from 127.21.201.65:37963 (error 108)
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":2412,"thread_start_us":2213,"threads_started":1}
Aug 12 00:59:44 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:59:44 dist-test-slave-7f01 krb5kdc[4074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.201.66@KRBTEST.COM for oryx/127.21.201.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:59:44.456583 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:44.447294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38217 (local address 127.21.201.66:38585)
0812 00:59:44.447951 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:44.447980 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:44.448141 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:44.448574 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:44.448581 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:44.449231 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:44.449484 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:44.449499 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:44.450522 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:44.450526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:44.451470 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:44.451481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:44.452312 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:44.452320 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:44.452497 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:44.453236 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:44.453255 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:44.456304 (+ 3049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.201.126:38217: Server oryx/127.21.201.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":303,"threads_started":1}
W20260812 00:59:44.456758 6015 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:44.447495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:38585 (local address 127.21.201.126:38217)
0812 00:59:44.447738 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:44.447745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:44.448121 (+ 376us) server_negotiation.cc:407] Connection header received
0812 00:59:44.448302 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:44.448308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:44.448392 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:44.448533 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:44.449635 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:44.450388 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:44.451588 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:44.452003 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:44.452096 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:44.456535 (+ 4439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.201.66:38585: BlockingRecv error: recv got EOF from 127.21.201.66:38585 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260812 00:59:44.776754 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 5096
I20260812 00:59:44.790230 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:44.777881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45030 (local address 127.21.201.65:38357)
0812 00:59:44.778617 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:44.778625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:44.778653 (+ 28us) server_negotiation.cc:407] Connection header received
0812 00:59:44.778717 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:44.778723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:44.778960 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:44.779120 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:44.780741 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:44.781798 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:44.783108 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:44.783665 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:44.783857 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:44.786427 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:44.786437 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:44.786453 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:44.786491 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:44.789463 (+ 2972us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.21.201.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0812 00:59:44.789870 (+ 407us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45030: Request ticket server kudu/127.21.201.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":374,"threads_started":1}
W20260812 00:59:44.790346 6020 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45030: Request ticket server kudu/127.21.201.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260812 00:59:44.790488 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 5096: Not authorized: Client connection negotiation failed: client connection to 127.21.201.65:38357: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.21.201.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260812 00:59:44.790724 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 5096
I20260812 00:59:44.819797 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 5385
I20260812 00:59:44.833770 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:44.821165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33850 (local address 127.21.201.66:38357)
0812 00:59:44.821892 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:44.821900 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:44.821933 (+ 33us) server_negotiation.cc:407] Connection header received
0812 00:59:44.821999 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:44.822006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:44.822228 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:44.822385 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:44.824045 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:44.825152 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:44.826341 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:44.826928 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:44.827040 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:44.829654 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:44.829663 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:44.829678 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:44.829711 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:44.833001 (+ 3290us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.21.201.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0812 00:59:44.833433 (+ 432us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33850: Request ticket server kudu/127.21.201.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":383,"threads_started":1}
W20260812 00:59:44.833894 6021 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33850: Request ticket server kudu/127.21.201.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260812 00:59:44.833973 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 5385: Not authorized: Client connection negotiation failed: client connection to 127.21.201.66:38357: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.21.201.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260812 00:59:44.834149 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 5385
I20260812 00:59:44.859956 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 5633
I20260812 00:59:44.874068 6023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:44.861075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43318 (local address 127.21.201.67:38443)
0812 00:59:44.861679 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:44.861686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:44.861712 (+ 26us) server_negotiation.cc:407] Connection header received
0812 00:59:44.861788 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:44.861795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:44.862024 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:44.862175 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:44.863625 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:44.864968 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:44.866199 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:44.866858 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:44.866975 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:44.870501 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:44.870527 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:44.870551 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:44.870616 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:44.873376 (+ 2760us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.21.201.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0812 00:59:44.873768 (+ 392us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43318: Request ticket server kudu/127.21.201.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":314,"threads_started":1}
W20260812 00:59:44.874185 6023 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43318: Request ticket server kudu/127.21.201.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260812 00:59:44.874233 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 5633: Not authorized: Client connection negotiation failed: client connection to 127.21.201.67:38443: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.21.201.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260812 00:59:44.874439 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 5633
I20260812 00:59:44.903697 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 4969
I20260812 00:59:44.905339 5085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:45.032107 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 4969
2026-08-12T00:59:45Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (22672 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-12T00:59:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:59:45Z Disabled control of system clock
I20260812 00:59:45.136854 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:37691
--webserver_interface=127.21.201.126
--webserver_port=0
--builtin_ntp_servers=127.21.201.84:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:37691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 00:59:45.495239 6032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:45.495577 6032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:45.495664 6032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:45.503228 6032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:59:45.503371 6032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:45.503410 6032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:45.503461 6032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:59:45.503509 6032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:59:45.513095 6032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:37691
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:37691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:45.514541 6032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:45.516388 6032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:45.529357 6042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:45.529277 6043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:45.529331 6046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:45.529803 6032 server_base.cc:1034] running on GCE node
I20260812 00:59:45.530930 6032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:45.532581 6032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:45.533839 6032 hybrid_clock.cc:648] HybridClock initialized: now 1786496385533792 us; error 40 us; skew 500 ppm
I20260812 00:59:45.534238 6032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 00:59:45.540174 6032 webserver.cc:460] Webserver started at http://127.21.201.126:39749/ using document root <none> and password file <none>
I20260812 00:59:45.541045 6032 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:45.541193 6032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:45.541560 6032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:59:45.544768 6032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "b88a6b3b00bb49d38b9ba68db6dbdd44"
format_stamp: "Formatted at 2026-08-12 00:59:45 on dist-test-slave-7f01"
server_key: "f0fa3ee894137bafa5805be7ba17df6f"
server_key_iv: "edd4d996be582687fa2ea6778723d6fa"
server_key_version: "encryptionkey@0"
I20260812 00:59:45.545639 6032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "b88a6b3b00bb49d38b9ba68db6dbdd44"
format_stamp: "Formatted at 2026-08-12 00:59:45 on dist-test-slave-7f01"
server_key: "f0fa3ee894137bafa5805be7ba17df6f"
server_key_iv: "edd4d996be582687fa2ea6778723d6fa"
server_key_version: "encryptionkey@0"
I20260812 00:59:45.551221 6032 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 00:59:45.554966 6064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:45.556555 6032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 00:59:45.556787 6032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "b88a6b3b00bb49d38b9ba68db6dbdd44"
format_stamp: "Formatted at 2026-08-12 00:59:45 on dist-test-slave-7f01"
server_key: "f0fa3ee894137bafa5805be7ba17df6f"
server_key_iv: "edd4d996be582687fa2ea6778723d6fa"
server_key_version: "encryptionkey@0"
I20260812 00:59:45.557049 6032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:45.580523 6032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:45.581583 6032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:45.581920 6032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:45.601929 6032 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:37691
I20260812 00:59:45.601951 6157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:37691 every 8 connection(s)
I20260812 00:59:45.603396 6032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 00:59:45.606810 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 6032
I20260812 00:59:45.607033 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 00:59:45.607925 6159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:45.607985 22309 external_mini_cluster.cc:1428] Setting key dad014c2be3951858faa71cd903df545
I20260812 00:59:45.622401 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:45.610271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47220 (local address 127.21.201.126:37691)
0812 00:59:45.611364 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:45.611375 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:45.611420 (+ 45us) server_negotiation.cc:407] Connection header received
0812 00:59:45.613084 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:45.613107 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:45.613593 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:45.613923 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:45.614676 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:45.616133 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:45.617927 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:45.618562 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:45.618702 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:45.620070 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:45.620078 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:45.620093 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:59:45.620667 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:45.620741 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:59:45.620762 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:45.620849 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:45.621096 (+ 247us) server_negotiation.cc:299] Negotiation successful
0812 00:59:45.621439 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":418,"threads_started":1}
I20260812 00:59:45.622619 6159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b88a6b3b00bb49d38b9ba68db6dbdd44: Bootstrap starting.
I20260812 00:59:45.626235 6159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b88a6b3b00bb49d38b9ba68db6dbdd44: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:45.627286 6159 log.cc:824] T 00000000000000000000000000000000 P b88a6b3b00bb49d38b9ba68db6dbdd44: Log is configured to *not* fsync() on all Append() calls
I20260812 00:59:45.630061 6159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b88a6b3b00bb49d38b9ba68db6dbdd44: No bootstrap required, opened a new log
I20260812 00:59:45.637975 6159 raft_consensus.cc:348] T 00000000000000000000000000000000 P b88a6b3b00bb49d38b9ba68db6dbdd44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88a6b3b00bb49d38b9ba68db6dbdd44" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 37691 } }
I20260812 00:59:45.638391 6159 raft_consensus.cc:374] T 00000000000000000000000000000000 P b88a6b3b00bb49d38b9ba68db6dbdd44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:45.638504 6159 raft_consensus.cc:729] T 00000000000000000000000000000000 P b88a6b3b00bb49d38b9ba68db6dbdd44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b88a6b3b00bb49d38b9ba68db6dbdd44, State: Initialized, Role: FOLLOWER
I20260812 00:59:45.639132 6159 consensus_queue.cc:260] T 00000000000000000000000000000000 P b88a6b3b00bb49d38b9ba68db6dbdd44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88a6b3b00bb49d38b9ba68db6dbdd44" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 37691 } }
I20260812 00:59:45.639403 6159 raft_consensus.cc:388] T 00000000000000000000000000000000 P b88a6b3b00bb49d38b9ba68db6dbdd44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:59:45.639564 6159 raft_consensus.cc:482] T 00000000000000000000000000000000 P b88a6b3b00bb49d38b9ba68db6dbdd44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:59:45.639786 6159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b88a6b3b00bb49d38b9ba68db6dbdd44 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:45.642180 6159 raft_consensus.cc:504] T 00000000000000000000000000000000 P b88a6b3b00bb49d38b9ba68db6dbdd44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88a6b3b00bb49d38b9ba68db6dbdd44" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 37691 } }
I20260812 00:59:45.642701 6159 leader_election.cc:302] T 00000000000000000000000000000000 P b88a6b3b00bb49d38b9ba68db6dbdd44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b88a6b3b00bb49d38b9ba68db6dbdd44; no voters:
I20260812 00:59:45.643388 6159 leader_election.cc:288] T 00000000000000000000000000000000 P b88a6b3b00bb49d38b9ba68db6dbdd44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:59:45.643595 6165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b88a6b3b00bb49d38b9ba68db6dbdd44 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:59:45.644647 6165 raft_consensus.cc:686] T 00000000000000000000000000000000 P b88a6b3b00bb49d38b9ba68db6dbdd44 [term 1 LEADER]: Becoming Leader. State: Replica: b88a6b3b00bb49d38b9ba68db6dbdd44, State: Running, Role: LEADER
I20260812 00:59:45.645192 6165 consensus_queue.cc:237] T 00000000000000000000000000000000 P b88a6b3b00bb49d38b9ba68db6dbdd44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88a6b3b00bb49d38b9ba68db6dbdd44" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 37691 } }
I20260812 00:59:45.645730 6159 sys_catalog.cc:564] T 00000000000000000000000000000000 P b88a6b3b00bb49d38b9ba68db6dbdd44 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:59:45.648283 6169 sys_catalog.cc:455] T 00000000000000000000000000000000 P b88a6b3b00bb49d38b9ba68db6dbdd44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b88a6b3b00bb49d38b9ba68db6dbdd44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88a6b3b00bb49d38b9ba68db6dbdd44" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 37691 } } }
I20260812 00:59:45.648528 6171 sys_catalog.cc:455] T 00000000000000000000000000000000 P b88a6b3b00bb49d38b9ba68db6dbdd44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b88a6b3b00bb49d38b9ba68db6dbdd44. Latest consensus state: current_term: 1 leader_uuid: "b88a6b3b00bb49d38b9ba68db6dbdd44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88a6b3b00bb49d38b9ba68db6dbdd44" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 37691 } } }
I20260812 00:59:45.648756 6169 sys_catalog.cc:458] T 00000000000000000000000000000000 P b88a6b3b00bb49d38b9ba68db6dbdd44 [sys.catalog]: This master's current role is: LEADER
I20260812 00:59:45.648835 6171 sys_catalog.cc:458] T 00000000000000000000000000000000 P b88a6b3b00bb49d38b9ba68db6dbdd44 [sys.catalog]: This master's current role is: LEADER
I20260812 00:59:45.651463 6181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:59:45.656376 6181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:59:45.665889 6181 catalog_manager.cc:1347] Generated new cluster ID: 4a6b95dc160144c98526891a5adbd781
I20260812 00:59:45.666045 6181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:59:45.679636 6181 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:59:45.681766 6181 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:59:45.692840 6181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b88a6b3b00bb49d38b9ba68db6dbdd44: Generated new TSK 0
I20260812 00:59:45.693985 6181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:59:45.712495 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:0
--local_ip_for_outbound_sockets=127.21.201.65
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:37691
--builtin_ntp_servers=127.21.201.84:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 00:59:46.084578 6198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:46.084896 6198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:46.084990 6198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:46.092768 6198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:46.092902 6198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:46.093027 6198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 00:59:46.102756 6198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:37691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:46.104143 6198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:46.105997 6198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:46.120075 6212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:46.120230 6209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:46.120155 6210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:46.121371 6198 server_base.cc:1034] running on GCE node
I20260812 00:59:46.122041 6198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:46.123137 6198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:46.124349 6198 hybrid_clock.cc:648] HybridClock initialized: now 1786496386124301 us; error 45 us; skew 500 ppm
I20260812 00:59:46.124680 6198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 00:59:46.127257 6198 webserver.cc:460] Webserver started at http://127.21.201.65:43483/ using document root <none> and password file <none>
I20260812 00:59:46.128057 6198 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:46.128154 6198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:46.128444 6198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:59:46.131616 6198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/data/instance:
uuid: "84c33056568c4373b61b5a722f1e6678"
format_stamp: "Formatted at 2026-08-12 00:59:46 on dist-test-slave-7f01"
server_key: "0081b501cc0a71e8ab06887773a1bf0c"
server_key_iv: "af9cbdc37dc2787c8cdf2e6e800ec16f"
server_key_version: "encryptionkey@0"
I20260812 00:59:46.132599 6198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance:
uuid: "84c33056568c4373b61b5a722f1e6678"
format_stamp: "Formatted at 2026-08-12 00:59:46 on dist-test-slave-7f01"
server_key: "0081b501cc0a71e8ab06887773a1bf0c"
server_key_iv: "af9cbdc37dc2787c8cdf2e6e800ec16f"
server_key_version: "encryptionkey@0"
I20260812 00:59:46.138170 6198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 00:59:46.141773 6223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:46.143189 6198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260812 00:59:46.143390 6198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "84c33056568c4373b61b5a722f1e6678"
format_stamp: "Formatted at 2026-08-12 00:59:46 on dist-test-slave-7f01"
server_key: "0081b501cc0a71e8ab06887773a1bf0c"
server_key_iv: "af9cbdc37dc2787c8cdf2e6e800ec16f"
server_key_version: "encryptionkey@0"
I20260812 00:59:46.143639 6198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:46.162938 6198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:46.163951 6198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:46.164283 6198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:46.165360 6198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:59:46.167722 6198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:59:46.167958 6198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260812 00:59:46.168114 6198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:59:46.168208 6198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:46.204109 6198 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:33469
I20260812 00:59:46.204166 6418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:33469 every 8 connection(s)
I20260812 00:59:46.205493 6198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 00:59:46.214473 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 6198
I20260812 00:59:46.214686 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance
I20260812 00:59:46.215361 22309 external_mini_cluster.cc:1428] Setting key 2aab9f2be6205bc2812ca25d598b9526
I20260812 00:59:46.218487 6422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:46.208220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:60731 (local address 127.21.201.126:37691)
0812 00:59:46.208767 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:46.208771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:46.209749 (+ 978us) server_negotiation.cc:407] Connection header received
0812 00:59:46.211599 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:46.211605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:46.211712 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:46.211889 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:46.212998 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:46.213731 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:46.215505 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:46.216002 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:46.216144 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:46.217690 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:46.217695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:46.217698 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:59:46.217716 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:46.217760 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:59:46.217773 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:46.217847 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:46.218120 (+ 273us) server_negotiation.cc:299] Negotiation successful
0812 00:59:46.218202 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":295,"threads_started":1}
I20260812 00:59:46.218816 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:0
--local_ip_for_outbound_sockets=127.21.201.66
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:37691
--builtin_ntp_servers=127.21.201.84:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 00:59:46.220207 6423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:46.208677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:37691 (local address 127.21.201.65:60731)
0812 00:59:46.209589 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:46.209632 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:46.211321 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:46.212127 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:46.212135 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:46.212219 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:46.212804 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:46.212821 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:46.213893 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:46.213925 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:46.215309 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:46.215322 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:46.216251 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:46.216258 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:46.216533 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:46.217330 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:59:46.217333 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:46.217373 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:59:46.217464 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:46.217880 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:46.217892 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:46.217900 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 00:59:46.218106 (+ 206us) client_negotiation.cc:240] Negotiation successful
0812 00:59:46.218591 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":284,"threads_started":1}
I20260812 00:59:46.221781 6420 heartbeater.cc:347] Connected to a master server at 127.21.201.126:37691
I20260812 00:59:46.222111 6420 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:46.222931 6420 heartbeater.cc:511] Master 127.21.201.126:37691 requested a full tablet report, sending...
I20260812 00:59:46.225144 6099 ts_manager.cc:194] Registered new tserver with Master: 84c33056568c4373b61b5a722f1e6678 (127.21.201.65:33469)
I20260812 00:59:46.226665 6099 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.201.65:60731
W20260812 00:59:46.573629 6428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:46.573944 6428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:46.574035 6428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:46.581576 6428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:46.581694 6428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:46.581813 6428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 00:59:46.591554 6428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:37691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:46.592967 6428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:46.594767 6428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:46.616513 6440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:46.617257 6437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:46.617648 6436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:46.619328 6428 server_base.cc:1034] running on GCE node
I20260812 00:59:46.620172 6428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:46.621501 6428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:46.622882 6428 hybrid_clock.cc:648] HybridClock initialized: now 1786496386622749 us; error 111 us; skew 500 ppm
I20260812 00:59:46.623378 6428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 00:59:46.626832 6428 webserver.cc:460] Webserver started at http://127.21.201.66:36807/ using document root <none> and password file <none>
I20260812 00:59:46.627885 6428 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:46.628031 6428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:46.628458 6428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:59:46.632969 6428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/data/instance:
uuid: "f2059b6e6fff4fa487ea9891ae5dce38"
format_stamp: "Formatted at 2026-08-12 00:59:46 on dist-test-slave-7f01"
server_key: "bad5f5680d080a471ce6a2be77132913"
server_key_iv: "525d21d6df71decc95daa3875fa4167d"
server_key_version: "encryptionkey@0"
I20260812 00:59:46.634099 6428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance:
uuid: "f2059b6e6fff4fa487ea9891ae5dce38"
format_stamp: "Formatted at 2026-08-12 00:59:46 on dist-test-slave-7f01"
server_key: "bad5f5680d080a471ce6a2be77132913"
server_key_iv: "525d21d6df71decc95daa3875fa4167d"
server_key_version: "encryptionkey@0"
I20260812 00:59:46.644519 6428 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.009s sys 0.000s
I20260812 00:59:46.650287 6450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:46.652406 6428 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 00:59:46.652622 6428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "f2059b6e6fff4fa487ea9891ae5dce38"
format_stamp: "Formatted at 2026-08-12 00:59:46 on dist-test-slave-7f01"
server_key: "bad5f5680d080a471ce6a2be77132913"
server_key_iv: "525d21d6df71decc95daa3875fa4167d"
server_key_version: "encryptionkey@0"
I20260812 00:59:46.652930 6428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:46.673027 6428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:46.673987 6428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:46.674324 6428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:46.675388 6428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:59:46.677000 6428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:59:46.677089 6428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:46.677250 6428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:59:46.677321 6428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:46.713701 6428 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:45319
I20260812 00:59:46.713760 6665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:45319 every 8 connection(s)
I20260812 00:59:46.715058 6428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 00:59:46.721484 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 6428
I20260812 00:59:46.721856 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance
I20260812 00:59:46.722566 22309 external_mini_cluster.cc:1428] Setting key 90ffdf422722206d36cc88945d390339
I20260812 00:59:46.726624 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:0
--local_ip_for_outbound_sockets=127.21.201.67
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:37691
--builtin_ntp_servers=127.21.201.84:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 00:59:46.731316 6422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:46.717860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:50945 (local address 127.21.201.126:37691)
0812 00:59:46.718068 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:46.718073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:46.719404 (+ 1331us) server_negotiation.cc:407] Connection header received
0812 00:59:46.722117 (+ 2713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:46.722122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:46.722202 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:46.722291 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:46.723830 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:46.725411 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:46.728187 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:46.728618 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:46.728720 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:46.730478 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:46.730484 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:46.730487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:59:46.730507 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:46.730542 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:59:46.730555 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:46.730637 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:46.730959 (+ 322us) server_negotiation.cc:299] Negotiation successful
0812 00:59:46.731050 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 00:59:46.732880 6677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:46.718328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:37691 (local address 127.21.201.66:50945)
0812 00:59:46.719252 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:46.719289 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:46.721627 (+ 2338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:46.722599 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:46.722613 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:46.722744 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:46.723595 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:46.723621 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:46.725719 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:46.725725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:46.728003 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:46.728024 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:46.728873 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:46.728884 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:46.729255 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:46.730077 (+ 822us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:59:46.730082 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:46.730139 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:59:46.730251 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:46.730665 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:46.730677 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:46.730688 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 00:59:46.730898 (+ 210us) client_negotiation.cc:240] Negotiation successful
0812 00:59:46.731343 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":305,"threads_started":1}
I20260812 00:59:46.734493 6667 heartbeater.cc:347] Connected to a master server at 127.21.201.126:37691
I20260812 00:59:46.734963 6667 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:46.736104 6667 heartbeater.cc:511] Master 127.21.201.126:37691 requested a full tablet report, sending...
I20260812 00:59:46.737696 6099 ts_manager.cc:194] Registered new tserver with Master: f2059b6e6fff4fa487ea9891ae5dce38 (127.21.201.66:45319)
I20260812 00:59:46.739019 6099 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.201.66:50945
W20260812 00:59:47.082345 6680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:47.082648 6680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:47.082736 6680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:47.090341 6680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:47.090456 6680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:47.090570 6680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 00:59:47.100114 6680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:37691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:47.101438 6680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:47.103204 6680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:47.117106 6688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:47.117130 6690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:47.117071 6693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:47.117950 6680 server_base.cc:1034] running on GCE node
I20260812 00:59:47.118589 6680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:47.119501 6680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:47.120736 6680 hybrid_clock.cc:648] HybridClock initialized: now 1786496387120660 us; error 57 us; skew 500 ppm
I20260812 00:59:47.121102 6680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 00:59:47.123363 6680 webserver.cc:460] Webserver started at http://127.21.201.67:32981/ using document root <none> and password file <none>
I20260812 00:59:47.124168 6680 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:47.124289 6680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:47.124634 6680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:59:47.127893 6680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/data/instance:
uuid: "a4615010fead4aedad077bf06197edc5"
format_stamp: "Formatted at 2026-08-12 00:59:47 on dist-test-slave-7f01"
server_key: "2796dd1f2e75ebf45adc5fad0ebff7ab"
server_key_iv: "92bf14dd920cab2e6872327d201816c1"
server_key_version: "encryptionkey@0"
I20260812 00:59:47.128760 6680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance:
uuid: "a4615010fead4aedad077bf06197edc5"
format_stamp: "Formatted at 2026-08-12 00:59:47 on dist-test-slave-7f01"
server_key: "2796dd1f2e75ebf45adc5fad0ebff7ab"
server_key_iv: "92bf14dd920cab2e6872327d201816c1"
server_key_version: "encryptionkey@0"
I20260812 00:59:47.134236 6680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 00:59:47.137768 6706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:47.139454 6680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 00:59:47.139621 6680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "a4615010fead4aedad077bf06197edc5"
format_stamp: "Formatted at 2026-08-12 00:59:47 on dist-test-slave-7f01"
server_key: "2796dd1f2e75ebf45adc5fad0ebff7ab"
server_key_iv: "92bf14dd920cab2e6872327d201816c1"
server_key_version: "encryptionkey@0"
I20260812 00:59:47.139940 6680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:47.155258 6680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:47.156236 6680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:47.156514 6680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:47.157470 6680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:59:47.159238 6680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:59:47.159354 6680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:47.159446 6680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:59:47.159504 6680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:47.197130 6680 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:35193
I20260812 00:59:47.197181 6921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:35193 every 8 connection(s)
I20260812 00:59:47.198557 6680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 00:59:47.199474 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 6680
I20260812 00:59:47.199697 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance
I20260812 00:59:47.200279 22309 external_mini_cluster.cc:1428] Setting key 0dbcf735045fc1de70f675872495dd81
I20260812 00:59:47.214406 6422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:47.202166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:58481 (local address 127.21.201.126:37691)
0812 00:59:47.202402 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:47.202408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:47.205160 (+ 2752us) server_negotiation.cc:407] Connection header received
0812 00:59:47.207018 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:47.207024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:47.207114 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:47.207228 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:47.208409 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:47.209547 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:47.211200 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:47.211621 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:47.211729 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:47.213428 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:47.213436 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:47.213440 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:59:47.213471 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:47.213539 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:59:47.213558 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:47.213696 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:47.214006 (+ 310us) server_negotiation.cc:299] Negotiation successful
0812 00:59:47.214122 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 00:59:47.215396 6925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:47.202697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:37691 (local address 127.21.201.67:58481)
0812 00:59:47.204944 (+ 2247us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:47.205014 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:47.206710 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:47.207439 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:47.207446 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:47.207530 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:47.208228 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:47.208246 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:47.209731 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:47.209735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:47.211028 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:47.211041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:47.211910 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:47.211916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:47.212135 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:47.212916 (+ 781us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:59:47.212918 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:47.212961 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:59:47.213054 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:47.213703 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:47.213711 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:47.213717 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 00:59:47.213876 (+ 159us) client_negotiation.cc:240] Negotiation successful
0812 00:59:47.214210 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1667,"thread_start_us":1400,"threads_started":1}
I20260812 00:59:47.217120 6923 heartbeater.cc:347] Connected to a master server at 127.21.201.126:37691
I20260812 00:59:47.217496 6923 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:47.218500 6923 heartbeater.cc:511] Master 127.21.201.126:37691 requested a full tablet report, sending...
I20260812 00:59:47.220110 6096 ts_manager.cc:194] Registered new tserver with Master: a4615010fead4aedad077bf06197edc5 (127.21.201.67:35193)
I20260812 00:59:47.221051 6096 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.201.67:58481
I20260812 00:59:47.229642 6420 heartbeater.cc:503] Master 127.21.201.126:37691 was elected leader, sending a full tablet report...
I20260812 00:59:47.232148 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 00:59:47.239056 6422 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:47.237283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32904 (local address 127.21.201.126:37691)
0812 00:59:47.237582 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:47.237592 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:47.238008 (+ 416us) server_negotiation.cc:407] Connection header received
0812 00:59:47.238169 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:47.238174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:47.238261 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:47.238354 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:47.238828 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32904: BlockingRecv error: recv got EOF from 127.0.0.1:32904 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 00:59:47.239818 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 6198
I20260812 00:59:47.248538 6938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:47.240906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38080 (local address 127.21.201.65:33469)
0812 00:59:47.241468 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:47.241475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:47.241500 (+ 25us) server_negotiation.cc:407] Connection header received
0812 00:59:47.241560 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:47.241566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:47.241802 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:47.241992 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:47.242451 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:47.243496 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:47.244833 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:47.245527 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:47.245674 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:47.246963 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:47.246972 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:47.246986 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:59:47.247622 (+ 636us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:47.247669 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:59:47.247689 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:47.247857 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:47.248000 (+ 143us) server_negotiation.cc:299] Negotiation successful
0812 00:59:47.248149 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":240,"threads_started":1}
I20260812 00:59:47.249920 6408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:47.336126 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 6198
I20260812 00:59:47.361547 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 6428
I20260812 00:59:47.369580 6943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:47.362663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57454 (local address 127.21.201.66:45319)
0812 00:59:47.363214 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:47.363221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:47.363249 (+ 28us) server_negotiation.cc:407] Connection header received
0812 00:59:47.363320 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:47.363326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:47.363542 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:47.363676 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:47.364110 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:47.365114 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:47.366315 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:47.366884 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:47.366990 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:47.368122 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:47.368129 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:47.368143 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:59:47.368740 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:47.368788 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:59:47.368813 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:47.368904 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:47.369044 (+ 140us) server_negotiation.cc:299] Negotiation successful
0812 00:59:47.369213 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":291,"threads_started":1}
I20260812 00:59:47.370875 6655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:47.457741 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 6428
I20260812 00:59:47.485280 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 6680
I20260812 00:59:47.493990 6945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:47.486382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50780 (local address 127.21.201.67:35193)
0812 00:59:47.487055 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:47.487063 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:47.487094 (+ 31us) server_negotiation.cc:407] Connection header received
0812 00:59:47.487164 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:47.487171 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:47.487388 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:47.487587 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:47.488028 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:47.489099 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:47.490411 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:47.491001 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:47.491158 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:47.492421 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:47.492430 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:47.492443 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:59:47.493074 (+ 631us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:47.493123 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:59:47.493149 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:47.493287 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:47.493446 (+ 159us) server_negotiation.cc:299] Negotiation successful
0812 00:59:47.493615 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":353,"threads_started":1}
I20260812 00:59:47.495450 6915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:47.584856 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 6680
I20260812 00:59:47.608891 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 6032
I20260812 00:59:47.610219 6153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:47.720889 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 6032
2026-08-12T00:59:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2681 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-12T00:59:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:59:47Z Disabled control of system clock
I20260812 00:59:47.817332 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:37767
--webserver_interface=127.21.201.126
--webserver_port=0
--builtin_ntp_servers=127.21.201.84:38429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:37767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 00:59:48.172777 6953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:48.173113 6953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:48.173272 6953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:48.180771 6953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:59:48.180912 6953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:48.181002 6953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:48.181125 6953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:59:48.181191 6953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:59:48.190812 6953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:38429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:37767
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:37767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:48.192293 6953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:48.194247 6953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:48.206570 6967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:48.206642 6969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:48.206672 6962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:48.207909 6953 server_base.cc:1034] running on GCE node
I20260812 00:59:48.208715 6953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:48.210187 6953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:48.211439 6953 hybrid_clock.cc:648] HybridClock initialized: now 1786496388211394 us; error 41 us; skew 500 ppm
I20260812 00:59:48.211881 6953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 00:59:48.214843 6953 webserver.cc:460] Webserver started at http://127.21.201.126:34943/ using document root <none> and password file <none>
I20260812 00:59:48.215605 6953 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:48.215785 6953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:48.216143 6953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:59:48.219169 6953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "82501cf411ca4ef59ad6c15f3ce08a64"
format_stamp: "Formatted at 2026-08-12 00:59:48 on dist-test-slave-7f01"
server_key: "c1dedeaf805d68428670eccb325fa970"
server_key_iv: "cc4e9542482ea873e4a9f920963a7c27"
server_key_version: "encryptionkey@0"
I20260812 00:59:48.220099 6953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "82501cf411ca4ef59ad6c15f3ce08a64"
format_stamp: "Formatted at 2026-08-12 00:59:48 on dist-test-slave-7f01"
server_key: "c1dedeaf805d68428670eccb325fa970"
server_key_iv: "cc4e9542482ea873e4a9f920963a7c27"
server_key_version: "encryptionkey@0"
I20260812 00:59:48.225478 6953 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 00:59:48.229156 6979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:48.230540 6953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 00:59:48.230754 6953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "82501cf411ca4ef59ad6c15f3ce08a64"
format_stamp: "Formatted at 2026-08-12 00:59:48 on dist-test-slave-7f01"
server_key: "c1dedeaf805d68428670eccb325fa970"
server_key_iv: "cc4e9542482ea873e4a9f920963a7c27"
server_key_version: "encryptionkey@0"
I20260812 00:59:48.231011 6953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:48.237207 6953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:48.238027 6953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:48.238354 6953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:48.260555 6953 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:37767
I20260812 00:59:48.260596 7091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:37767 every 8 connection(s)
I20260812 00:59:48.262063 6953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 00:59:48.265350 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 6953
I20260812 00:59:48.265590 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 00:59:48.266036 7092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:48.266304 22309 external_mini_cluster.cc:1428] Setting key ebf4f485aa774268ac5ac6e11875835a
I20260812 00:59:48.274840 7097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:48.268462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57892 (local address 127.21.201.126:37767)
0812 00:59:48.269525 (+ 1063us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:48.269535 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:48.269579 (+ 44us) server_negotiation.cc:407] Connection header received
0812 00:59:48.271244 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:48.271264 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:48.271671 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:48.272140 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:48.273259 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:48.273264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:48.273276 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:59:48.273293 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:48.273350 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:59:48.273364 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:48.273621 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:48.273758 (+ 137us) server_negotiation.cc:299] Negotiation successful
0812 00:59:48.273984 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":403,"threads_started":1}
I20260812 00:59:48.280049 7092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82501cf411ca4ef59ad6c15f3ce08a64: Bootstrap starting.
I20260812 00:59:48.283366 7092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82501cf411ca4ef59ad6c15f3ce08a64: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:48.284525 7092 log.cc:824] T 00000000000000000000000000000000 P 82501cf411ca4ef59ad6c15f3ce08a64: Log is configured to *not* fsync() on all Append() calls
I20260812 00:59:48.287530 7092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82501cf411ca4ef59ad6c15f3ce08a64: No bootstrap required, opened a new log
I20260812 00:59:48.295384 7092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82501cf411ca4ef59ad6c15f3ce08a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82501cf411ca4ef59ad6c15f3ce08a64" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 37767 } }
I20260812 00:59:48.295835 7092 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82501cf411ca4ef59ad6c15f3ce08a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:48.295964 7092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82501cf411ca4ef59ad6c15f3ce08a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82501cf411ca4ef59ad6c15f3ce08a64, State: Initialized, Role: FOLLOWER
I20260812 00:59:48.296540 7092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82501cf411ca4ef59ad6c15f3ce08a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82501cf411ca4ef59ad6c15f3ce08a64" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 37767 } }
I20260812 00:59:48.296794 7092 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82501cf411ca4ef59ad6c15f3ce08a64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:59:48.296953 7092 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82501cf411ca4ef59ad6c15f3ce08a64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:59:48.297137 7092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82501cf411ca4ef59ad6c15f3ce08a64 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:48.299413 7092 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82501cf411ca4ef59ad6c15f3ce08a64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82501cf411ca4ef59ad6c15f3ce08a64" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 37767 } }
I20260812 00:59:48.300040 7092 leader_election.cc:302] T 00000000000000000000000000000000 P 82501cf411ca4ef59ad6c15f3ce08a64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82501cf411ca4ef59ad6c15f3ce08a64; no voters:
I20260812 00:59:48.300669 7092 leader_election.cc:288] T 00000000000000000000000000000000 P 82501cf411ca4ef59ad6c15f3ce08a64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:59:48.300861 7101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82501cf411ca4ef59ad6c15f3ce08a64 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:59:48.301683 7101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82501cf411ca4ef59ad6c15f3ce08a64 [term 1 LEADER]: Becoming Leader. State: Replica: 82501cf411ca4ef59ad6c15f3ce08a64, State: Running, Role: LEADER
I20260812 00:59:48.302374 7101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82501cf411ca4ef59ad6c15f3ce08a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82501cf411ca4ef59ad6c15f3ce08a64" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 37767 } }
I20260812 00:59:48.303098 7092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82501cf411ca4ef59ad6c15f3ce08a64 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:59:48.305512 7103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82501cf411ca4ef59ad6c15f3ce08a64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82501cf411ca4ef59ad6c15f3ce08a64. Latest consensus state: current_term: 1 leader_uuid: "82501cf411ca4ef59ad6c15f3ce08a64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82501cf411ca4ef59ad6c15f3ce08a64" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 37767 } } }
I20260812 00:59:48.306066 7103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82501cf411ca4ef59ad6c15f3ce08a64 [sys.catalog]: This master's current role is: LEADER
I20260812 00:59:48.305994 7102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82501cf411ca4ef59ad6c15f3ce08a64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82501cf411ca4ef59ad6c15f3ce08a64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82501cf411ca4ef59ad6c15f3ce08a64" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 37767 } } }
I20260812 00:59:48.306345 7102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82501cf411ca4ef59ad6c15f3ce08a64 [sys.catalog]: This master's current role is: LEADER
I20260812 00:59:48.307968 7113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:59:48.312737 7113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:59:48.321703 7113 catalog_manager.cc:1347] Generated new cluster ID: f45469661c3b49c1812b6393a947705f
I20260812 00:59:48.321830 7113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:59:48.336717 7113 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:59:48.337265 7113 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:59:48.345909 7113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82501cf411ca4ef59ad6c15f3ce08a64: Generated new TSK 0
I20260812 00:59:48.346984 7113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:59:48.359148 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:0
--local_ip_for_outbound_sockets=127.21.201.65
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:37767
--builtin_ntp_servers=127.21.201.84:38429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 00:59:48.716776 7132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:48.717221 7132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:48.717407 7132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:48.725505 7132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:48.725669 7132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:48.725867 7132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 00:59:48.736017 7132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:38429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:37767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:48.737532 7132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:48.739430 7132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:48.753120 7141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:48.753157 7140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:48.753152 7149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:48.754078 7132 server_base.cc:1034] running on GCE node
I20260812 00:59:48.754755 7132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:48.755837 7132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:48.757169 7132 hybrid_clock.cc:648] HybridClock initialized: now 1786496388757113 us; error 37 us; skew 500 ppm
I20260812 00:59:48.757611 7132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 00:59:48.760448 7132 webserver.cc:460] Webserver started at http://127.21.201.65:45771/ using document root <none> and password file <none>
I20260812 00:59:48.761165 7132 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:48.761255 7132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:48.761530 7132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:59:48.764554 7132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/data/instance:
uuid: "d35f8b994d7b4a2dbb41f612edb18f4c"
format_stamp: "Formatted at 2026-08-12 00:59:48 on dist-test-slave-7f01"
server_key: "a885d88a725a9b4384de034fcee416d6"
server_key_iv: "34586cb60188b8688cb945757847a754"
server_key_version: "encryptionkey@0"
I20260812 00:59:48.765341 7132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance:
uuid: "d35f8b994d7b4a2dbb41f612edb18f4c"
format_stamp: "Formatted at 2026-08-12 00:59:48 on dist-test-slave-7f01"
server_key: "a885d88a725a9b4384de034fcee416d6"
server_key_iv: "34586cb60188b8688cb945757847a754"
server_key_version: "encryptionkey@0"
I20260812 00:59:48.770845 7132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 00:59:48.774775 7158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:48.776441 7132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 00:59:48.776639 7132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "d35f8b994d7b4a2dbb41f612edb18f4c"
format_stamp: "Formatted at 2026-08-12 00:59:48 on dist-test-slave-7f01"
server_key: "a885d88a725a9b4384de034fcee416d6"
server_key_iv: "34586cb60188b8688cb945757847a754"
server_key_version: "encryptionkey@0"
I20260812 00:59:48.776895 7132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:48.782967 7132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:48.783892 7132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:48.784194 7132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:48.785161 7132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:59:48.786856 7132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:59:48.787009 7132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:48.787144 7132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:59:48.787410 7132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:48.826795 7132 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:45993
I20260812 00:59:48.826853 7363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:45993 every 8 connection(s)
I20260812 00:59:48.828259 7132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 00:59:48.830250 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 7132
I20260812 00:59:48.830482 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance
I20260812 00:59:48.831274 22309 external_mini_cluster.cc:1428] Setting key 82aff2a05870b169aef42965e4ce3cfc
I20260812 00:59:48.834394 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:0
--local_ip_for_outbound_sockets=127.21.201.66
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:37767
--builtin_ntp_servers=127.21.201.84:38429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 00:59:48.837566 7367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:48.831067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:57603 (local address 127.21.201.126:37767)
0812 00:59:48.831647 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:48.831654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:48.833015 (+ 1361us) server_negotiation.cc:407] Connection header received
0812 00:59:48.834943 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:48.834952 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:48.835056 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:48.835180 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:48.836682 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:48.836691 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:48.836698 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:59:48.836722 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:48.836782 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:59:48.836800 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:48.836958 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:48.837108 (+ 150us) server_negotiation.cc:299] Negotiation successful
0812 00:59:48.837180 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":297,"threads_started":1}
I20260812 00:59:48.838415 7370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:48.831638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:37767 (local address 127.21.201.65:57603)
0812 00:59:48.832826 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:48.832887 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:48.834646 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:48.835422 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:48.835445 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:48.835543 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:48.836364 (+ 821us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:59:48.836367 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:48.836406 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:59:48.836503 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:48.836907 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:48.836915 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:48.836920 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:59:48.837143 (+ 223us) client_negotiation.cc:240] Negotiation successful
0812 00:59:48.837349 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":356,"threads_started":1}
I20260812 00:59:48.840513 7364 heartbeater.cc:347] Connected to a master server at 127.21.201.126:37767
I20260812 00:59:48.840847 7364 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:48.841496 7364 heartbeater.cc:511] Master 127.21.201.126:37767 requested a full tablet report, sending...
I20260812 00:59:48.843966 7011 ts_manager.cc:194] Registered new tserver with Master: d35f8b994d7b4a2dbb41f612edb18f4c (127.21.201.65:45993)
W20260812 00:59:49.191107 7371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:49.191424 7371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:49.191591 7371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:49.200165 7371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:49.200304 7371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:49.200515 7371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 00:59:49.210915 7371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:38429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:37767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:49.212448 7371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:49.214673 7371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:49.228472 7384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:49.228511 7382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:49.228578 7394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:49.229481 7371 server_base.cc:1034] running on GCE node
I20260812 00:59:49.230122 7371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:49.231122 7371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:49.232384 7371 hybrid_clock.cc:648] HybridClock initialized: now 1786496389232336 us; error 53 us; skew 500 ppm
I20260812 00:59:49.232756 7371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 00:59:49.235795 7371 webserver.cc:460] Webserver started at http://127.21.201.66:34325/ using document root <none> and password file <none>
I20260812 00:59:49.236574 7371 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:49.236722 7371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:49.237083 7371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:59:49.240316 7371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/data/instance:
uuid: "9f55a71b4e73474cb9792adb0857d58d"
format_stamp: "Formatted at 2026-08-12 00:59:49 on dist-test-slave-7f01"
server_key: "2e11a4bba54d075c9779868864534cd5"
server_key_iv: "6467b5d452882c0ff2092982d8e71813"
server_key_version: "encryptionkey@0"
I20260812 00:59:49.241165 7371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance:
uuid: "9f55a71b4e73474cb9792adb0857d58d"
format_stamp: "Formatted at 2026-08-12 00:59:49 on dist-test-slave-7f01"
server_key: "2e11a4bba54d075c9779868864534cd5"
server_key_iv: "6467b5d452882c0ff2092982d8e71813"
server_key_version: "encryptionkey@0"
I20260812 00:59:49.246948 7371 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260812 00:59:49.250918 7406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:49.252503 7371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 00:59:49.252707 7371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "9f55a71b4e73474cb9792adb0857d58d"
format_stamp: "Formatted at 2026-08-12 00:59:49 on dist-test-slave-7f01"
server_key: "2e11a4bba54d075c9779868864534cd5"
server_key_iv: "6467b5d452882c0ff2092982d8e71813"
server_key_version: "encryptionkey@0"
I20260812 00:59:49.252976 7371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:49.259152 7371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:49.260064 7371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:49.260349 7371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:49.261343 7371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:59:49.262989 7371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:59:49.263119 7371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:49.263263 7371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:59:49.263365 7371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:49.301287 7371 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:40791
I20260812 00:59:49.301350 7623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:40791 every 8 connection(s)
I20260812 00:59:49.302697 7371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 00:59:49.307704 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 7371
I20260812 00:59:49.307952 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance
I20260812 00:59:49.308547 22309 external_mini_cluster.cc:1428] Setting key 043b8e918f672d76bd53aca24e7966ff
I20260812 00:59:49.311973 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:0
--local_ip_for_outbound_sockets=127.21.201.67
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:37767
--builtin_ntp_servers=127.21.201.84:38429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 00:59:49.312487 7367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:49.305595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:44367 (local address 127.21.201.126:37767)
0812 00:59:49.305818 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:49.305824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:49.307371 (+ 1547us) server_negotiation.cc:407] Connection header received
0812 00:59:49.309861 (+ 2490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:49.309867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:49.309961 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:49.310119 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:49.311636 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:49.311640 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:49.311644 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:59:49.311660 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:49.311693 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:59:49.311704 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:49.311889 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:49.312158 (+ 269us) server_negotiation.cc:299] Negotiation successful
0812 00:59:49.312239 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 00:59:49.313836 7626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:49.306172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:37767 (local address 127.21.201.66:44367)
0812 00:59:49.307170 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:49.307212 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:49.309550 (+ 2338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:49.310354 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:49.310363 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:49.310504 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:49.311265 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:59:49.311269 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:49.311321 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:59:49.311438 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:49.311876 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:49.311890 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:49.311898 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 00:59:49.312230 (+ 332us) client_negotiation.cc:240] Negotiation successful
0812 00:59:49.312497 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":296,"threads_started":1}
I20260812 00:59:49.315263 7624 heartbeater.cc:347] Connected to a master server at 127.21.201.126:37767
I20260812 00:59:49.315593 7624 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:49.316283 7624 heartbeater.cc:511] Master 127.21.201.126:37767 requested a full tablet report, sending...
I20260812 00:59:49.317754 7011 ts_manager.cc:194] Registered new tserver with Master: 9f55a71b4e73474cb9792adb0857d58d (127.21.201.66:40791)
W20260812 00:59:49.661168 7629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:49.661502 7629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:49.661659 7629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:49.669257 7629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:49.669384 7629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:49.669566 7629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 00:59:49.679037 7629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:38429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:37767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:49.680555 7629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:49.682456 7629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:49.695987 7640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:49.696012 7637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:49.695925 7638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:49.696990 7629 server_base.cc:1034] running on GCE node
I20260812 00:59:49.697654 7629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:49.698660 7629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:49.699882 7629 hybrid_clock.cc:648] HybridClock initialized: now 1786496389699815 us; error 61 us; skew 500 ppm
I20260812 00:59:49.700217 7629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 00:59:49.702718 7629 webserver.cc:460] Webserver started at http://127.21.201.67:35421/ using document root <none> and password file <none>
I20260812 00:59:49.703501 7629 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:49.703637 7629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:49.704028 7629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:59:49.707183 7629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/data/instance:
uuid: "5031724738e746ff82401ec93caa4ca3"
format_stamp: "Formatted at 2026-08-12 00:59:49 on dist-test-slave-7f01"
server_key: "c8bdc5724716bca9ddd21e9b1c8302c6"
server_key_iv: "41e7b12576bfd1433ca3ef5a37103221"
server_key_version: "encryptionkey@0"
I20260812 00:59:49.708106 7629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance:
uuid: "5031724738e746ff82401ec93caa4ca3"
format_stamp: "Formatted at 2026-08-12 00:59:49 on dist-test-slave-7f01"
server_key: "c8bdc5724716bca9ddd21e9b1c8302c6"
server_key_iv: "41e7b12576bfd1433ca3ef5a37103221"
server_key_version: "encryptionkey@0"
I20260812 00:59:49.713560 7629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 00:59:49.717085 7652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:49.718473 7629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 00:59:49.718670 7629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "5031724738e746ff82401ec93caa4ca3"
format_stamp: "Formatted at 2026-08-12 00:59:49 on dist-test-slave-7f01"
server_key: "c8bdc5724716bca9ddd21e9b1c8302c6"
server_key_iv: "41e7b12576bfd1433ca3ef5a37103221"
server_key_version: "encryptionkey@0"
I20260812 00:59:49.718928 7629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:49.725035 7629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:49.725903 7629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:49.726203 7629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:49.727196 7629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:59:49.728802 7629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:59:49.728938 7629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:49.729081 7629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:59:49.729173 7629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:49.765152 7629 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:36149
I20260812 00:59:49.765210 7861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:36149 every 8 connection(s)
I20260812 00:59:49.766544 7629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 00:59:49.774700 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 7629
I20260812 00:59:49.774917 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance
I20260812 00:59:49.775104 7367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:49.769173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:44531 (local address 127.21.201.126:37767)
0812 00:59:49.769393 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:49.769400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:49.770847 (+ 1447us) server_negotiation.cc:407] Connection header received
0812 00:59:49.772648 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:49.772653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:49.772712 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:49.772804 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:49.774124 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:49.774129 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:49.774132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:59:49.774170 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:49.774212 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:59:49.774243 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:49.774382 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:49.774673 (+ 291us) server_negotiation.cc:299] Negotiation successful
0812 00:59:49.774761 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 00:59:49.775497 22309 external_mini_cluster.cc:1428] Setting key e297ef586d3c9683f7f834b136a928ec
I20260812 00:59:49.776638 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:49.769759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:37767 (local address 127.21.201.67:44531)
0812 00:59:49.770679 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:49.770715 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:49.772395 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:49.773020 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:49.773028 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:49.773116 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:49.773771 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:59:49.773774 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:49.773829 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:59:49.773925 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:49.774335 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:49.774377 (+ 42us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:49.774387 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 00:59:49.775430 (+ 1043us) client_negotiation.cc:240] Negotiation successful
0812 00:59:49.775687 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":293,"threads_started":1}
I20260812 00:59:49.778090 7862 heartbeater.cc:347] Connected to a master server at 127.21.201.126:37767
I20260812 00:59:49.778442 7862 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:49.779091 7862 heartbeater.cc:511] Master 127.21.201.126:37767 requested a full tablet report, sending...
I20260812 00:59:49.780637 7006 ts_manager.cc:194] Registered new tserver with Master: 5031724738e746ff82401ec93caa4ca3 (127.21.201.67:36149)
I20260812 00:59:49.791404 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 00:59:49.799009 7367 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:59:49.797191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57908 (local address 127.21.201.126:37767)
0812 00:59:49.797399 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:49.797404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:49.797877 (+ 473us) server_negotiation.cc:407] Connection header received
0812 00:59:49.798034 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:49.798038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:49.798094 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:49.798190 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:49.798793 (+ 603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57908: BlockingRecv error: recv got EOF from 127.0.0.1:57908 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 00:59:49.800177 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 7132
I20260812 00:59:49.804999 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:49.801299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59048 (local address 127.21.201.65:45993)
0812 00:59:49.801918 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:49.801926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:49.802349 (+ 423us) server_negotiation.cc:407] Connection header received
0812 00:59:49.802467 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:49.802474 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:49.802707 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:49.802911 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:49.804069 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:49.804077 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:49.804089 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:59:49.804110 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:49.804155 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:59:49.804170 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:49.804307 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:49.804580 (+ 273us) server_negotiation.cc:299] Negotiation successful
0812 00:59:49.804694 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":323,"threads_started":1}
I20260812 00:59:49.806283 7354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:49.893743 7364 heartbeater.cc:503] Master 127.21.201.126:37767 was elected leader, sending a full tablet report...
I20260812 00:59:49.895109 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 7132
W20260812 00:59:49.922988 6983 connection.cc:535] server connection from 127.21.201.65:57603 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 00:59:49.923213 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 7371
I20260812 00:59:49.927006 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:49.924253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39306 (local address 127.21.201.66:40791)
0812 00:59:49.924780 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:49.924789 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:49.924817 (+ 28us) server_negotiation.cc:407] Connection header received
0812 00:59:49.924885 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:49.924891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:49.925116 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:49.925297 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:49.926285 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:49.926291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:49.926303 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:59:49.926315 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:49.926357 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:59:49.926374 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:49.926538 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:49.926618 (+ 80us) server_negotiation.cc:299] Negotiation successful
0812 00:59:49.926727 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":253,"threads_started":1}
I20260812 00:59:49.928442 7615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:50.012477 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 7371
I20260812 00:59:50.036854 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 7629
I20260812 00:59:50.041374 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:50.038009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40474 (local address 127.21.201.67:36149)
0812 00:59:50.038680 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:50.038687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:50.038719 (+ 32us) server_negotiation.cc:407] Connection header received
0812 00:59:50.038785 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:50.038791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:50.039023 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:50.039198 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:50.040538 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:50.040544 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:50.040556 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:59:50.040574 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:50.040621 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:59:50.040638 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:50.040850 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:50.040929 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 00:59:50.041052 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":338,"threads_started":1}
I20260812 00:59:50.042712 7854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:50.127894 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 7629
I20260812 00:59:50.151393 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 6953
I20260812 00:59:50.152822 7082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:50.260824 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 6953
2026-08-12T00:59:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2544 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:59:50 dist-test-slave-7f01 krb5kdc[7900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:59:50 dist-test-slave-7f01 krb5kdc[7900](info): set up 2 sockets
Aug 12 00:59:50 dist-test-slave-7f01 krb5kdc[7900](info): commencing operation
krb5kdc: starting...
W20260812 00:59:51.648159 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.332s user 0.000s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 00:59:51 dist-test-slave-7f01 krb5kdc[7900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:59:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:59:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 00:59:51.961853 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:40509
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:44465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:40509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:59:52.304555 7920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:52.305063 7920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:52.305250 7920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:52.312901 7920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:59:52.313084 7920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:52.313179 7920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:52.313267 7920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:59:52.313344 7920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:59:52.323413 7920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:44465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:40509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:40509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:52.324900 7920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:52.326259 7920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:52.338454 7927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:52.339133 7934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:52.339815 7937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:52.340332 7920 server_base.cc:1034] running on GCE node
I20260812 00:59:52.341183 7920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:52.342664 7920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:52.343858 7920 hybrid_clock.cc:648] HybridClock initialized: now 1786496392343814 us; error 34 us; skew 500 ppm
I20260812 00:59:52.344233 7920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:52 dist-test-slave-7f01 krb5kdc[7900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496392, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:52.356189 7920 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 00:59:52.358117 7920 webserver.cc:460] Webserver started at http://127.21.201.126:38097/ using document root <none> and password file <none>
I20260812 00:59:52.358915 7920 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:52.359081 7920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:52.359434 7920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:59:52.362803 7920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "7624bb4df12f43f78a18b3934487d0ca"
format_stamp: "Formatted at 2026-08-12 00:59:52 on dist-test-slave-7f01"
server_key: "fde9cb2bf0768d5ea05e8d10801df205"
server_key_iv: "d8f42a54bfcfc9073b329050b3ab107f"
server_key_version: "encryptionkey@0"
I20260812 00:59:52.363629 7920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "7624bb4df12f43f78a18b3934487d0ca"
format_stamp: "Formatted at 2026-08-12 00:59:52 on dist-test-slave-7f01"
server_key: "fde9cb2bf0768d5ea05e8d10801df205"
server_key_iv: "d8f42a54bfcfc9073b329050b3ab107f"
server_key_version: "encryptionkey@0"
I20260812 00:59:52.369251 7920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260812 00:59:52.373111 7954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:52.374426 7920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 00:59:52.374634 7920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "7624bb4df12f43f78a18b3934487d0ca"
format_stamp: "Formatted at 2026-08-12 00:59:52 on dist-test-slave-7f01"
server_key: "fde9cb2bf0768d5ea05e8d10801df205"
server_key_iv: "d8f42a54bfcfc9073b329050b3ab107f"
server_key_version: "encryptionkey@0"
I20260812 00:59:52.374850 7920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:52.386998 7920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:52.391356 7920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:52.391659 7920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:52.411298 7920 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:40509
I20260812 00:59:52.411340 8050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:40509 every 8 connection(s)
I20260812 00:59:52.412796 7920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 00:59:52.416692 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:52.421429 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 7920
I20260812 00:59:52.421684 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 00:59:52.422545 22309 external_mini_cluster.cc:1428] Setting key d7c3e101da5ca7748a74a73aaa37d82f
I20260812 00:59:52.429101 8051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7624bb4df12f43f78a18b3934487d0ca: Bootstrap starting.
I20260812 00:59:52.433102 8051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7624bb4df12f43f78a18b3934487d0ca: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:52.434413 8051 log.cc:824] T 00000000000000000000000000000000 P 7624bb4df12f43f78a18b3934487d0ca: Log is configured to *not* fsync() on all Append() calls
I20260812 00:59:52.437517 8051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7624bb4df12f43f78a18b3934487d0ca: No bootstrap required, opened a new log
Aug 12 00:59:52 dist-test-slave-7f01 krb5kdc[7900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:59:52.446189 8051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7624bb4df12f43f78a18b3934487d0ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7624bb4df12f43f78a18b3934487d0ca" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 40509 } }
I20260812 00:59:52.446725 8051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7624bb4df12f43f78a18b3934487d0ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:52.446894 8051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7624bb4df12f43f78a18b3934487d0ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7624bb4df12f43f78a18b3934487d0ca, State: Initialized, Role: FOLLOWER
I20260812 00:59:52.447515 8051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7624bb4df12f43f78a18b3934487d0ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7624bb4df12f43f78a18b3934487d0ca" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 40509 } }
I20260812 00:59:52.447795 8051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7624bb4df12f43f78a18b3934487d0ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:59:52.447917 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:52.424736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39822 (local address 127.21.201.126:40509)
0812 00:59:52.426090 (+ 1354us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:52.426103 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:52.426157 (+ 54us) server_negotiation.cc:407] Connection header received
0812 00:59:52.428009 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:52.428031 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:52.428429 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:52.428758 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:52.430782 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:52.432241 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:52.434074 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:52.434826 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:52.435007 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:52.439293 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:52.439304 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:52.439349 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:52.439413 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:52.443110 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:52.443942 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:52.443949 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:52.443956 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:52.444137 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:52.444565 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:52.444577 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:52.444579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:52.445184 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:52.445533 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:52.445977 (+ 444us) server_negotiation.cc:299] Negotiation successful
0812 00:59:52.446448 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1084,"thread_start_us":725,"threads_started":1}
I20260812 00:59:52.447975 8051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7624bb4df12f43f78a18b3934487d0ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:59:52.448328 8051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7624bb4df12f43f78a18b3934487d0ca [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:52.451362 8051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7624bb4df12f43f78a18b3934487d0ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7624bb4df12f43f78a18b3934487d0ca" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 40509 } }
I20260812 00:59:52.451998 8051 leader_election.cc:302] T 00000000000000000000000000000000 P 7624bb4df12f43f78a18b3934487d0ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7624bb4df12f43f78a18b3934487d0ca; no voters:
I20260812 00:59:52.452600 8051 leader_election.cc:288] T 00000000000000000000000000000000 P 7624bb4df12f43f78a18b3934487d0ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:59:52.452796 8062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7624bb4df12f43f78a18b3934487d0ca [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:59:52.453509 8062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7624bb4df12f43f78a18b3934487d0ca [term 1 LEADER]: Becoming Leader. State: Replica: 7624bb4df12f43f78a18b3934487d0ca, State: Running, Role: LEADER
I20260812 00:59:52.454200 8062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7624bb4df12f43f78a18b3934487d0ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7624bb4df12f43f78a18b3934487d0ca" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 40509 } }
I20260812 00:59:52.455026 8051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7624bb4df12f43f78a18b3934487d0ca [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:59:52.457746 8066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7624bb4df12f43f78a18b3934487d0ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7624bb4df12f43f78a18b3934487d0ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7624bb4df12f43f78a18b3934487d0ca" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 40509 } } }
I20260812 00:59:52.457568 8069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7624bb4df12f43f78a18b3934487d0ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7624bb4df12f43f78a18b3934487d0ca. Latest consensus state: current_term: 1 leader_uuid: "7624bb4df12f43f78a18b3934487d0ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7624bb4df12f43f78a18b3934487d0ca" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 40509 } } }
I20260812 00:59:52.458132 8066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7624bb4df12f43f78a18b3934487d0ca [sys.catalog]: This master's current role is: LEADER
I20260812 00:59:52.458191 8069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7624bb4df12f43f78a18b3934487d0ca [sys.catalog]: This master's current role is: LEADER
I20260812 00:59:52.459189 8084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:59:52.467020 8084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:59:52.477486 8084 catalog_manager.cc:1347] Generated new cluster ID: 1920345e5c734b3680233505b6ddde3f
I20260812 00:59:52.477630 8084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:59:52.510921 8084 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:59:52.512408 8084 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:59:52.518181 8084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7624bb4df12f43f78a18b3934487d0ca: Generated new TSK 0
I20260812 00:59:52.519256 8084 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.201.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
I20260812 00:59:52.597949 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:0
--local_ip_for_outbound_sockets=127.21.201.65
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:40509
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:44465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:59:52.960042 8103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:52.960409 8103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:52.960520 8103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:52.968163 8103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:52.968297 8103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:52.968421 8103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 00:59:52.978844 8103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:44465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:40509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:52.980417 8103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:52.981799 8103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:52.996029 8119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:52.996089 8124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:52.996083 8114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:52.997437 8103 server_base.cc:1034] running on GCE node
I20260812 00:59:52.998558 8103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:52.999713 8103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:53.001021 8103 hybrid_clock.cc:648] HybridClock initialized: now 1786496393000973 us; error 39 us; skew 500 ppm
I20260812 00:59:53.001417 8103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:53 dist-test-slave-7f01 krb5kdc[7900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496393, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:53.004953 8103 init.cc:377] Logged in from keytab as kudu/127.21.201.65@KRBTEST.COM (short username kudu)
I20260812 00:59:53.006654 8103 webserver.cc:460] Webserver started at http://127.21.201.65:40293/ using document root <none> and password file <none>
I20260812 00:59:53.007408 8103 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:53.007520 8103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:53.007853 8103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:59:53.010886 8103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/data/instance:
uuid: "69ce59fd38574c69bf5147c41ae8cfa0"
format_stamp: "Formatted at 2026-08-12 00:59:53 on dist-test-slave-7f01"
server_key: "fba0b5f931105af36d6b4a25d84bff68"
server_key_iv: "b8fb6337f0dd9b826b996fa95575de8b"
server_key_version: "encryptionkey@0"
I20260812 00:59:53.011653 8103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance:
uuid: "69ce59fd38574c69bf5147c41ae8cfa0"
format_stamp: "Formatted at 2026-08-12 00:59:53 on dist-test-slave-7f01"
server_key: "fba0b5f931105af36d6b4a25d84bff68"
server_key_iv: "b8fb6337f0dd9b826b996fa95575de8b"
server_key_version: "encryptionkey@0"
I20260812 00:59:53.017216 8103 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 00:59:53.021410 8136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:53.022985 8103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260812 00:59:53.023178 8103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "69ce59fd38574c69bf5147c41ae8cfa0"
format_stamp: "Formatted at 2026-08-12 00:59:53 on dist-test-slave-7f01"
server_key: "fba0b5f931105af36d6b4a25d84bff68"
server_key_iv: "b8fb6337f0dd9b826b996fa95575de8b"
server_key_version: "encryptionkey@0"
I20260812 00:59:53.023392 8103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:53.053323 8103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:53.059916 8103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:53.060322 8103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:53.061582 8103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:59:53.063547 8103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:59:53.063656 8103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:53.063861 8103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:59:53.063947 8103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:53.102227 8103 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:46295
I20260812 00:59:53.102330 8356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:46295 every 8 connection(s)
I20260812 00:59:53.103607 8103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 00:59:53.109573 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 8103
I20260812 00:59:53.109841 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance
I20260812 00:59:53.110554 22309 external_mini_cluster.cc:1428] Setting key d18a9fd31b3a70d94741600ff261d542
Aug 12 00:59:53 dist-test-slave-7f01 krb5kdc[7900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496393, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:59:53.126019 8369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:53.106867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:47953 (local address 127.21.201.126:40509)
0812 00:59:53.107447 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:53.107452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:53.108575 (+ 1123us) server_negotiation.cc:407] Connection header received
0812 00:59:53.110603 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:53.110609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:53.110713 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:53.110843 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:53.112997 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:53.113756 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:53.115491 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:53.116000 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:53.116148 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:53.119887 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:53.119893 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:53.119896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:53.119931 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:53.122729 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:53.123548 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:53.123560 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:53.123563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:53.123694 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:53.124241 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:53.124248 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:53.124256 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:53.124540 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:53.124702 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:53.125538 (+ 836us) server_negotiation.cc:299] Negotiation successful
0812 00:59:53.125736 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":272,"threads_started":1}
I20260812 00:59:53.127621 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:53.107531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:40509 (local address 127.21.201.65:47953)
0812 00:59:53.108376 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:53.108425 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:53.110298 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:53.111136 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:53.111148 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:53.111900 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:53.112789 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:53.112828 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:53.113944 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:53.113950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:53.115326 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:53.115339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:53.116249 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:53.116259 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:53.116518 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:53.117268 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:53.117293 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:53.119616 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:53.122898 (+ 3282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:53.122909 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:53.122928 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:53.123404 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:53.123876 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:53.123882 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:53.123886 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:53.124112 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:53.124677 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:53.124685 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:53.125059 (+ 374us) client_negotiation.cc:769] Sending connection context
0812 00:59:53.125360 (+ 301us) client_negotiation.cc:240] Negotiation successful
0812 00:59:53.125990 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":384,"threads_started":1}
I20260812 00:59:53.129454 8364 heartbeater.cc:347] Connected to a master server at 127.21.201.126:40509
I20260812 00:59:53.129791 8364 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:53.130600 8364 heartbeater.cc:511] Master 127.21.201.126:40509 requested a full tablet report, sending...
I20260812 00:59:53.133225 7978 ts_manager.cc:194] Registered new tserver with Master: 69ce59fd38574c69bf5147c41ae8cfa0 (127.21.201.65:46295)
I20260812 00:59:53.134799 7978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.65@KRBTEST.COM'} at 127.21.201.65:47953
WARNING: no policy specified for kudu/127.21.201.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
I20260812 00:59:53.179628 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:0
--local_ip_for_outbound_sockets=127.21.201.66
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:40509
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:44465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:59:53.539018 8376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:53.539323 8376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:53.539520 8376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:53.547235 8376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:53.547363 8376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:53.547567 8376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 00:59:53.558203 8376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:44465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:40509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:53.559785 8376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:53.561439 8376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:59:53.576809 8388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:53.576836 8386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:53.576882 8393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:53.578317 8376 server_base.cc:1034] running on GCE node
I20260812 00:59:53.579134 8376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:53.580371 8376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:53.581614 8376 hybrid_clock.cc:648] HybridClock initialized: now 1786496393581567 us; error 31 us; skew 500 ppm
I20260812 00:59:53.582033 8376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:53 dist-test-slave-7f01 krb5kdc[7900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496393, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:53.585791 8376 init.cc:377] Logged in from keytab as kudu/127.21.201.66@KRBTEST.COM (short username kudu)
I20260812 00:59:53.587486 8376 webserver.cc:460] Webserver started at http://127.21.201.66:35049/ using document root <none> and password file <none>
I20260812 00:59:53.588310 8376 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:53.588466 8376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:53.588795 8376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:59:53.592043 8376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/data/instance:
uuid: "ddbe3f402cec4142b962542377361bc4"
format_stamp: "Formatted at 2026-08-12 00:59:53 on dist-test-slave-7f01"
server_key: "b2b2784f2cc9f65395663a3d7cfd9ce1"
server_key_iv: "8ef29b2b00ad9f134997c2a0610cecee"
server_key_version: "encryptionkey@0"
I20260812 00:59:53.592864 8376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance:
uuid: "ddbe3f402cec4142b962542377361bc4"
format_stamp: "Formatted at 2026-08-12 00:59:53 on dist-test-slave-7f01"
server_key: "b2b2784f2cc9f65395663a3d7cfd9ce1"
server_key_iv: "8ef29b2b00ad9f134997c2a0610cecee"
server_key_version: "encryptionkey@0"
I20260812 00:59:53.599186 8376 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 00:59:53.603811 8405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:53.605280 8376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 00:59:53.605502 8376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "ddbe3f402cec4142b962542377361bc4"
format_stamp: "Formatted at 2026-08-12 00:59:53 on dist-test-slave-7f01"
server_key: "b2b2784f2cc9f65395663a3d7cfd9ce1"
server_key_iv: "8ef29b2b00ad9f134997c2a0610cecee"
server_key_version: "encryptionkey@0"
I20260812 00:59:53.605743 8376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:53.622315 8376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:53.626974 8376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:53.627317 8376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:53.628540 8376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:59:53.630074 8376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:59:53.630185 8376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:53.630352 8376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:59:53.630429 8376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:53.668563 8376 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:33645
I20260812 00:59:53.668628 8627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:33645 every 8 connection(s)
I20260812 00:59:53.669966 8376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 00:59:53.672094 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 8376
I20260812 00:59:53.672318 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance
I20260812 00:59:53.672869 22309 external_mini_cluster.cc:1428] Setting key 9898526506e3dc79bf4c101756d7b6cb
Aug 12 00:59:53 dist-test-slave-7f01 krb5kdc[7900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496393, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:59:53.692517 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:53.672676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:47039 (local address 127.21.201.126:40509)
0812 00:59:53.673261 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:53.673270 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:53.674270 (+ 1000us) server_negotiation.cc:407] Connection header received
0812 00:59:53.676310 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:53.676316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:53.676418 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:53.676540 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:53.678202 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:53.679023 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:53.680799 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:53.681237 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:53.681385 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:53.685824 (+ 4439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:53.685830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:53.685833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:53.685869 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:53.689178 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:53.690018 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:53.690030 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:53.690033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:53.690166 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:53.690637 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:53.690642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:53.690644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:53.690937 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:53.691102 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:53.691854 (+ 752us) server_negotiation.cc:299] Negotiation successful
0812 00:59:53.692101 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":305,"threads_started":1}
I20260812 00:59:53.693553 8631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:53.673232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:40509 (local address 127.21.201.66:47039)
0812 00:59:53.674120 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:53.674165 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:53.676038 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:53.676790 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:53.676798 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:53.677331 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:53.678020 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:53.678050 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:53.679235 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:53.679239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:53.680641 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:53.680654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:53.681471 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:53.681478 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:53.681717 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:53.682475 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:53.682501 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:53.685547 (+ 3046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:53.689367 (+ 3820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:53.689378 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:53.689403 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:53.689876 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:53.690287 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:53.690293 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:53.690296 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:53.690504 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:53.691072 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:53.691080 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:53.691445 (+ 365us) client_negotiation.cc:769] Sending connection context
0812 00:59:53.691718 (+ 273us) client_negotiation.cc:240] Negotiation successful
0812 00:59:53.692339 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":336,"threads_started":1}
I20260812 00:59:53.695106 8628 heartbeater.cc:347] Connected to a master server at 127.21.201.126:40509
I20260812 00:59:53.695497 8628 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:53.696547 8628 heartbeater.cc:511] Master 127.21.201.126:40509 requested a full tablet report, sending...
I20260812 00:59:53.698446 7978 ts_manager.cc:194] Registered new tserver with Master: ddbe3f402cec4142b962542377361bc4 (127.21.201.66:33645)
I20260812 00:59:53.699870 7978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.66@KRBTEST.COM'} at 127.21.201.66:47039
WARNING: no policy specified for kudu/127.21.201.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
I20260812 00:59:53.747545 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:0
--local_ip_for_outbound_sockets=127.21.201.67
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:40509
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:44465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:59:54.112656 8634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:59:54.113003 8634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:59:54.113179 8634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:59:54.120972 8634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:59:54.121153 8634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:59:54.121336 8634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 00:59:54.133205 8634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:44465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:40509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 00:59:54.134721 8634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:59:54.136173 8634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:59:54.138095 8364 heartbeater.cc:503] Master 127.21.201.126:40509 was elected leader, sending a full tablet report...
W20260812 00:59:54.151721 8642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:54.151703 8641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:59:54.152117 8647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:59:54.152486 8634 server_base.cc:1034] running on GCE node
I20260812 00:59:54.153190 8634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:59:54.154263 8634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:59:54.155519 8634 hybrid_clock.cc:648] HybridClock initialized: now 1786496394155485 us; error 60 us; skew 500 ppm
I20260812 00:59:54.155931 8634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 00:59:54 dist-test-slave-7f01 krb5kdc[7900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496394, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:59:54.159497 8634 init.cc:377] Logged in from keytab as kudu/127.21.201.67@KRBTEST.COM (short username kudu)
I20260812 00:59:54.161170 8634 webserver.cc:460] Webserver started at http://127.21.201.67:36299/ using document root <none> and password file <none>
I20260812 00:59:54.161947 8634 fs_manager.cc:359] Metadata directory not provided
I20260812 00:59:54.162060 8634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:59:54.162328 8634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:59:54.165473 8634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/data/instance:
uuid: "134ca7183ef7455691a15a71077ce771"
format_stamp: "Formatted at 2026-08-12 00:59:54 on dist-test-slave-7f01"
server_key: "d8548a826fce45e1ecf9e27abd1aee59"
server_key_iv: "d3731c9032fef23a18601f00b4b93e69"
server_key_version: "encryptionkey@0"
I20260812 00:59:54.166266 8634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance:
uuid: "134ca7183ef7455691a15a71077ce771"
format_stamp: "Formatted at 2026-08-12 00:59:54 on dist-test-slave-7f01"
server_key: "d8548a826fce45e1ecf9e27abd1aee59"
server_key_iv: "d3731c9032fef23a18601f00b4b93e69"
server_key_version: "encryptionkey@0"
I20260812 00:59:54.172617 8634 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260812 00:59:54.177111 8663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:54.178812 8634 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260812 00:59:54.178997 8634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "134ca7183ef7455691a15a71077ce771"
format_stamp: "Formatted at 2026-08-12 00:59:54 on dist-test-slave-7f01"
server_key: "d8548a826fce45e1ecf9e27abd1aee59"
server_key_iv: "d3731c9032fef23a18601f00b4b93e69"
server_key_version: "encryptionkey@0"
I20260812 00:59:54.179189 8634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:59:54.197310 8634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:59:54.201964 8634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:59:54.202253 8634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:59:54.203190 8634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:59:54.204675 8634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:59:54.204751 8634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:54.204850 8634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:59:54.204897 8634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:59:54.240193 8634 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:37571
I20260812 00:59:54.240243 8876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:37571 every 8 connection(s)
I20260812 00:59:54.241550 8634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 00:59:54.249745 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 8634
I20260812 00:59:54.249974 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance
I20260812 00:59:54.250425 22309 external_mini_cluster.cc:1428] Setting key f27ea0a845e46fcbc6d3c8509730c473
Aug 12 00:59:54 dist-test-slave-7f01 krb5kdc[7900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496394, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 00:59:54.265558 8882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:54.244395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:55103 (local address 127.21.201.126:40509)
0812 00:59:54.244945 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:54.244950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:54.245799 (+ 849us) server_negotiation.cc:407] Connection header received
0812 00:59:54.247714 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:54.247719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:54.247836 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:54.247963 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:54.249569 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.250593 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.252733 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.253209 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.253314 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:54.258559 (+ 5245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:54.258564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:54.258568 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:54.258607 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:54.262036 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:54.262873 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:54.262878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:54.262880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:54.263017 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:54.263443 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:54.263447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:54.263449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:54.263831 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:54.263987 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:54.264858 (+ 871us) server_negotiation.cc:299] Negotiation successful
0812 00:59:54.265112 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":288,"threads_started":1}
I20260812 00:59:54.266799 8883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:54.244792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:40509 (local address 127.21.201.67:55103)
0812 00:59:54.245636 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:54.245676 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:54.247442 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:54.248206 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:54.248215 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:54.248732 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:59:54.249303 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:54.249377 (+ 74us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.250764 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.250772 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:54.252470 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:54.252488 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.253486 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.253496 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:54.253827 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:54.254736 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:59:54.254769 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:59:54.258318 (+ 3549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:59:54.262200 (+ 3882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:54.262208 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:54.262224 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:54.262711 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:54.263128 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:59:54.263131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:59:54.263134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:59:54.263332 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:59:54.263956 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:59:54.263962 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:59:54.264415 (+ 453us) client_negotiation.cc:769] Sending connection context
0812 00:59:54.264704 (+ 289us) client_negotiation.cc:240] Negotiation successful
0812 00:59:54.265285 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":391,"threads_started":1}
I20260812 00:59:54.268333 8878 heartbeater.cc:347] Connected to a master server at 127.21.201.126:40509
I20260812 00:59:54.268676 8878 heartbeater.cc:464] Registering TS with master...
I20260812 00:59:54.269496 8878 heartbeater.cc:511] Master 127.21.201.126:40509 requested a full tablet report, sending...
I20260812 00:59:54.271129 7978 ts_manager.cc:194] Registered new tserver with Master: 134ca7183ef7455691a15a71077ce771 (127.21.201.67:37571)
I20260812 00:59:54.272150 7978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.67@KRBTEST.COM'} at 127.21.201.67:55103
I20260812 00:59:54.280835 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:59:54.301090 8882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:54.286689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39824 (local address 127.21.201.126:40509)
0812 00:59:54.287015 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:54.287022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:54.287346 (+ 324us) server_negotiation.cc:407] Connection header received
0812 00:59:54.287499 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:54.287504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:54.287577 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:54.287665 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:54.289232 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.290063 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.291401 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.291968 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.292089 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:54.294580 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:54.294585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:54.294588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:54.294624 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:54.297698 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:54.298434 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:54.298438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:54.298441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:54.298567 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:54.299077 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:54.299084 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:54.299087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:54.299646 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:54.300072 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:54.300451 (+ 379us) server_negotiation.cc:299] Negotiation successful
0812 00:59:54.300706 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260812 00:59:54.305562 7978 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:59:54.313235 7978 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:59:54.353503 8903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:54.327399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33148 (local address 127.21.201.65:46295)
0812 00:59:54.342776 (+ 15377us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:54.342790 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:54.342834 (+ 44us) server_negotiation.cc:407] Connection header received
0812 00:59:54.343176 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:54.343186 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:54.343529 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:54.344381 (+ 852us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:54.346247 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.347951 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.350756 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.352166 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.352334 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:54.352600 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:54.352957 (+ 357us) server_negotiation.cc:299] Negotiation successful
0812 00:59:54.353140 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15114,"thread_start_us":14894,"threads_started":1}
I20260812 00:59:54.353825 8904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:54.327097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:46295 (local address 127.0.0.1:33148)
0812 00:59:54.342605 (+ 15508us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:54.342671 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:54.342870 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:54.343818 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:54.343825 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:54.345232 (+ 1407us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:54.345752 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:54.345768 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.348129 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.348135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:54.350550 (+ 2415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:54.350565 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.352420 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.352431 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:54.352666 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:54.352729 (+ 63us) client_negotiation.cc:769] Sending connection context
0812 00:59:54.352881 (+ 152us) client_negotiation.cc:240] Negotiation successful
0812 00:59:54.353094 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15193,"mutex_wait_us":72,"spinlock_wait_cycles":384,"thread_start_us":16252,"threads_started":1}
I20260812 00:59:54.355382 8902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:54.327635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:37571 (local address 127.0.0.1:41452)
0812 00:59:54.343725 (+ 16090us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:54.343767 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:54.343908 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:54.344894 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:54.344899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:54.344958 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:54.348423 (+ 3465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:54.348438 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.350332 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.350336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:54.352556 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:54.352571 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.354439 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.354446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:54.354633 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:54.354741 (+ 108us) client_negotiation.cc:769] Sending connection context
0812 00:59:54.354826 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 00:59:54.354965 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15859,"spinlock_wait_cycles":8064,"thread_start_us":14944,"threads_started":1}
I20260812 00:59:54.355417 8905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:54.328438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41452 (local address 127.21.201.67:37571)
0812 00:59:54.343710 (+ 15272us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:54.343718 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:54.344011 (+ 293us) server_negotiation.cc:407] Connection header received
0812 00:59:54.344185 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:54.344195 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:54.344541 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:54.345388 (+ 847us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:54.348650 (+ 3262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.350150 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.352745 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.354232 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.354377 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:54.354601 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:54.354900 (+ 299us) server_negotiation.cc:299] Negotiation successful
0812 00:59:54.355058 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15063,"thread_start_us":15272,"threads_started":1}
I20260812 00:59:54.357427 8906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:54.345939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:33645 (local address 127.0.0.1:59178)
0812 00:59:54.348014 (+ 2075us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:54.348034 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:54.348148 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:54.349335 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:54.349339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:54.349366 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:54.349807 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:54.349818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.351764 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.351770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:54.353931 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:54.353951 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.356688 (+ 2737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.356697 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:54.356830 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:54.356853 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 00:59:54.356970 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 00:59:54.357102 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1941,"thread_start_us":1795,"threads_started":1}
I20260812 00:59:54.357544 8907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:54.345993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59178 (local address 127.21.201.66:33645)
0812 00:59:54.348714 (+ 2721us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:54.348723 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:54.348750 (+ 27us) server_negotiation.cc:407] Connection header received
0812 00:59:54.348855 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:54.348863 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:54.349114 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:54.349330 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:54.350005 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.351586 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.354128 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.355643 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.355827 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:54.356033 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:54.357017 (+ 984us) server_negotiation.cc:299] Negotiation successful
0812 00:59:54.357188 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2512,"thread_start_us":2330,"threads_started":1}
I20260812 00:59:54.358414 8223 tablet_service.cc:1467] Processing CreateTablet for tablet 97d4b54eb8be4462aae2732709413844 (DEFAULT_TABLE table=test-table [id=bca3c332820d487fafc99593321b1ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:59:54.359366 8745 tablet_service.cc:1467] Processing CreateTablet for tablet 97d4b54eb8be4462aae2732709413844 (DEFAULT_TABLE table=test-table [id=bca3c332820d487fafc99593321b1ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:59:54.360349 8223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97d4b54eb8be4462aae2732709413844. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:54.360849 8745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97d4b54eb8be4462aae2732709413844. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:54.361063 8501 tablet_service.cc:1467] Processing CreateTablet for tablet 97d4b54eb8be4462aae2732709413844 (DEFAULT_TABLE table=test-table [id=bca3c332820d487fafc99593321b1ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:59:54.362272 8501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97d4b54eb8be4462aae2732709413844. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:59:54.373512 8909 tablet_bootstrap.cc:492] T 97d4b54eb8be4462aae2732709413844 P ddbe3f402cec4142b962542377361bc4: Bootstrap starting.
I20260812 00:59:54.376868 8909 tablet_bootstrap.cc:654] T 97d4b54eb8be4462aae2732709413844 P ddbe3f402cec4142b962542377361bc4: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:54.378001 8909 log.cc:824] T 97d4b54eb8be4462aae2732709413844 P ddbe3f402cec4142b962542377361bc4: Log is configured to *not* fsync() on all Append() calls
I20260812 00:59:54.380811 8911 tablet_bootstrap.cc:492] T 97d4b54eb8be4462aae2732709413844 P 69ce59fd38574c69bf5147c41ae8cfa0: Bootstrap starting.
I20260812 00:59:54.381727 8913 tablet_bootstrap.cc:492] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771: Bootstrap starting.
I20260812 00:59:54.382898 8909 tablet_bootstrap.cc:492] T 97d4b54eb8be4462aae2732709413844 P ddbe3f402cec4142b962542377361bc4: No bootstrap required, opened a new log
I20260812 00:59:54.383211 8909 ts_tablet_manager.cc:1397] T 97d4b54eb8be4462aae2732709413844 P ddbe3f402cec4142b962542377361bc4: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20260812 00:59:54.386629 8911 tablet_bootstrap.cc:654] T 97d4b54eb8be4462aae2732709413844 P 69ce59fd38574c69bf5147c41ae8cfa0: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:54.388007 8911 log.cc:824] T 97d4b54eb8be4462aae2732709413844 P 69ce59fd38574c69bf5147c41ae8cfa0: Log is configured to *not* fsync() on all Append() calls
I20260812 00:59:54.389292 8913 tablet_bootstrap.cc:654] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771: Neither blocks nor log segments found. Creating new log.
I20260812 00:59:54.390396 8913 log.cc:824] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771: Log is configured to *not* fsync() on all Append() calls
I20260812 00:59:54.391470 8911 tablet_bootstrap.cc:492] T 97d4b54eb8be4462aae2732709413844 P 69ce59fd38574c69bf5147c41ae8cfa0: No bootstrap required, opened a new log
I20260812 00:59:54.391826 8911 ts_tablet_manager.cc:1397] T 97d4b54eb8be4462aae2732709413844 P 69ce59fd38574c69bf5147c41ae8cfa0: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260812 00:59:54.393687 8913 tablet_bootstrap.cc:492] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771: No bootstrap required, opened a new log
I20260812 00:59:54.393968 8913 ts_tablet_manager.cc:1397] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260812 00:59:54.393992 8909 raft_consensus.cc:348] T 97d4b54eb8be4462aae2732709413844 P ddbe3f402cec4142b962542377361bc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ce59fd38574c69bf5147c41ae8cfa0" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 46295 } } peers { permanent_uuid: "134ca7183ef7455691a15a71077ce771" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37571 } } peers { permanent_uuid: "ddbe3f402cec4142b962542377361bc4" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33645 } }
I20260812 00:59:54.394462 8909 raft_consensus.cc:374] T 97d4b54eb8be4462aae2732709413844 P ddbe3f402cec4142b962542377361bc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:54.394577 8909 raft_consensus.cc:729] T 97d4b54eb8be4462aae2732709413844 P ddbe3f402cec4142b962542377361bc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddbe3f402cec4142b962542377361bc4, State: Initialized, Role: FOLLOWER
I20260812 00:59:54.395233 8909 consensus_queue.cc:260] T 97d4b54eb8be4462aae2732709413844 P ddbe3f402cec4142b962542377361bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ce59fd38574c69bf5147c41ae8cfa0" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 46295 } } peers { permanent_uuid: "134ca7183ef7455691a15a71077ce771" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37571 } } peers { permanent_uuid: "ddbe3f402cec4142b962542377361bc4" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33645 } }
I20260812 00:59:54.396466 8628 heartbeater.cc:503] Master 127.21.201.126:40509 was elected leader, sending a full tablet report...
I20260812 00:59:54.397089 8909 ts_tablet_manager.cc:1428] T 97d4b54eb8be4462aae2732709413844 P ddbe3f402cec4142b962542377361bc4: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260812 00:59:54.402494 8911 raft_consensus.cc:348] T 97d4b54eb8be4462aae2732709413844 P 69ce59fd38574c69bf5147c41ae8cfa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ce59fd38574c69bf5147c41ae8cfa0" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 46295 } } peers { permanent_uuid: "134ca7183ef7455691a15a71077ce771" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37571 } } peers { permanent_uuid: "ddbe3f402cec4142b962542377361bc4" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33645 } }
I20260812 00:59:54.402999 8911 raft_consensus.cc:374] T 97d4b54eb8be4462aae2732709413844 P 69ce59fd38574c69bf5147c41ae8cfa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:54.403121 8911 raft_consensus.cc:729] T 97d4b54eb8be4462aae2732709413844 P 69ce59fd38574c69bf5147c41ae8cfa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ce59fd38574c69bf5147c41ae8cfa0, State: Initialized, Role: FOLLOWER
I20260812 00:59:54.403846 8911 consensus_queue.cc:260] T 97d4b54eb8be4462aae2732709413844 P 69ce59fd38574c69bf5147c41ae8cfa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ce59fd38574c69bf5147c41ae8cfa0" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 46295 } } peers { permanent_uuid: "134ca7183ef7455691a15a71077ce771" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37571 } } peers { permanent_uuid: "ddbe3f402cec4142b962542377361bc4" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33645 } }
I20260812 00:59:54.403721 8913 raft_consensus.cc:348] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ce59fd38574c69bf5147c41ae8cfa0" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 46295 } } peers { permanent_uuid: "134ca7183ef7455691a15a71077ce771" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37571 } } peers { permanent_uuid: "ddbe3f402cec4142b962542377361bc4" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33645 } }
I20260812 00:59:54.404214 8913 raft_consensus.cc:374] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:59:54.404340 8913 raft_consensus.cc:729] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 134ca7183ef7455691a15a71077ce771, State: Initialized, Role: FOLLOWER
I20260812 00:59:54.404946 8913 consensus_queue.cc:260] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ce59fd38574c69bf5147c41ae8cfa0" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 46295 } } peers { permanent_uuid: "134ca7183ef7455691a15a71077ce771" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37571 } } peers { permanent_uuid: "ddbe3f402cec4142b962542377361bc4" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33645 } }
I20260812 00:59:54.405668 8911 ts_tablet_manager.cc:1428] T 97d4b54eb8be4462aae2732709413844 P 69ce59fd38574c69bf5147c41ae8cfa0: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260812 00:59:54.406085 8878 heartbeater.cc:503] Master 127.21.201.126:40509 was elected leader, sending a full tablet report...
I20260812 00:59:54.406759 8913 ts_tablet_manager.cc:1428] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
W20260812 00:59:54.422142 8629 tablet.cc:2369] T 97d4b54eb8be4462aae2732709413844 P ddbe3f402cec4142b962542377361bc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:59:54.493384 8881 tablet.cc:2369] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:59:54.591005 8925 raft_consensus.cc:482] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:59:54.591405 8925 raft_consensus.cc:504] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ce59fd38574c69bf5147c41ae8cfa0" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 46295 } } peers { permanent_uuid: "134ca7183ef7455691a15a71077ce771" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37571 } } peers { permanent_uuid: "ddbe3f402cec4142b962542377361bc4" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33645 } }
I20260812 00:59:54.593267 8925 leader_election.cc:288] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69ce59fd38574c69bf5147c41ae8cfa0 (127.21.201.65:46295), ddbe3f402cec4142b962542377361bc4 (127.21.201.66:33645)
I20260812 00:59:54.600927 8883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:54.593822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:46295 (local address 127.21.201.67:49879)
0812 00:59:54.594170 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:54.594193 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:54.594393 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:54.594965 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:54.594975 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:54.595002 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:54.595225 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:54.595238 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.597004 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.597008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:54.598790 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:54.598808 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.600300 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.600304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:54.600453 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:54.600487 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 00:59:54.600591 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 00:59:54.600693 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":4}
I20260812 00:59:54.601080 8903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:54.593935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:49879 (local address 127.21.201.65:46295)
0812 00:59:54.594172 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:54.594180 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:54.594295 (+ 115us) server_negotiation.cc:407] Connection header received
0812 00:59:54.594555 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:54.594558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:54.594645 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:54.594747 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:54.595476 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.596763 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.599040 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.600114 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.600301 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:54.600379 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:54.600656 (+ 277us) server_negotiation.cc:299] Negotiation successful
0812 00:59:54.600801 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 00:59:54.602034 8907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:54.594623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:35791 (local address 127.21.201.66:33645)
0812 00:59:54.594869 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:54.594875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:54.595262 (+ 387us) server_negotiation.cc:407] Connection header received
0812 00:59:54.595493 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:54.595499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:54.595596 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:54.595714 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:59:54.596762 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.597611 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.599814 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.601063 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.601256 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:54.601336 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:54.601616 (+ 280us) server_negotiation.cc:299] Negotiation successful
0812 00:59:54.601756 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 00:59:54.602067 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:54.594308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:33645 (local address 127.21.201.67:35791)
0812 00:59:54.595133 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0812 00:59:54.595157 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 00:59:54.595280 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:59:54.595983 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:59:54.595989 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:59:54.596035 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:59:54.596435 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:54.596446 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.597840 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.597845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:54.599565 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:59:54.599576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.601294 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.601300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:59:54.601447 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:54.601471 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 00:59:54.601585 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 00:59:54.601715 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":466,"threads_started":1}
I20260812 00:59:54.602429 8271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d4b54eb8be4462aae2732709413844" candidate_uuid: "134ca7183ef7455691a15a71077ce771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ce59fd38574c69bf5147c41ae8cfa0" is_pre_election: true
I20260812 00:59:54.603083 8271 raft_consensus.cc:2454] T 97d4b54eb8be4462aae2732709413844 P 69ce59fd38574c69bf5147c41ae8cfa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 134ca7183ef7455691a15a71077ce771 in term 0.
I20260812 00:59:54.603147 8537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d4b54eb8be4462aae2732709413844" candidate_uuid: "134ca7183ef7455691a15a71077ce771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddbe3f402cec4142b962542377361bc4" is_pre_election: true
I20260812 00:59:54.603695 8537 raft_consensus.cc:2454] T 97d4b54eb8be4462aae2732709413844 P ddbe3f402cec4142b962542377361bc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 134ca7183ef7455691a15a71077ce771 in term 0.
I20260812 00:59:54.603842 8668 leader_election.cc:302] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 134ca7183ef7455691a15a71077ce771, 69ce59fd38574c69bf5147c41ae8cfa0; no voters:
I20260812 00:59:54.604323 8925 raft_consensus.cc:2781] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:59:54.604477 8925 raft_consensus.cc:482] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:59:54.604607 8925 raft_consensus.cc:3037] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771 [term 0 FOLLOWER]: Advancing to term 1
W20260812 00:59:54.606562 8366 tablet.cc:2369] T 97d4b54eb8be4462aae2732709413844 P 69ce59fd38574c69bf5147c41ae8cfa0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:59:54.607446 8925 raft_consensus.cc:504] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ce59fd38574c69bf5147c41ae8cfa0" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 46295 } } peers { permanent_uuid: "134ca7183ef7455691a15a71077ce771" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37571 } } peers { permanent_uuid: "ddbe3f402cec4142b962542377361bc4" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33645 } }
I20260812 00:59:54.608110 8925 leader_election.cc:288] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771 [CANDIDATE]: Term 1 election: Requested vote from peers 69ce59fd38574c69bf5147c41ae8cfa0 (127.21.201.65:46295), ddbe3f402cec4142b962542377361bc4 (127.21.201.66:33645)
I20260812 00:59:54.609097 8271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d4b54eb8be4462aae2732709413844" candidate_uuid: "134ca7183ef7455691a15a71077ce771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ce59fd38574c69bf5147c41ae8cfa0"
I20260812 00:59:54.609194 8537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d4b54eb8be4462aae2732709413844" candidate_uuid: "134ca7183ef7455691a15a71077ce771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddbe3f402cec4142b962542377361bc4"
I20260812 00:59:54.609426 8271 raft_consensus.cc:3037] T 97d4b54eb8be4462aae2732709413844 P 69ce59fd38574c69bf5147c41ae8cfa0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:54.609428 8537 raft_consensus.cc:3037] T 97d4b54eb8be4462aae2732709413844 P ddbe3f402cec4142b962542377361bc4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:59:54.612138 8537 raft_consensus.cc:2454] T 97d4b54eb8be4462aae2732709413844 P ddbe3f402cec4142b962542377361bc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 134ca7183ef7455691a15a71077ce771 in term 1.
I20260812 00:59:54.612627 8668 leader_election.cc:302] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 134ca7183ef7455691a15a71077ce771, ddbe3f402cec4142b962542377361bc4; no voters:
I20260812 00:59:54.613071 8925 raft_consensus.cc:2781] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:59:54.613274 8271 raft_consensus.cc:2454] T 97d4b54eb8be4462aae2732709413844 P 69ce59fd38574c69bf5147c41ae8cfa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 134ca7183ef7455691a15a71077ce771 in term 1.
I20260812 00:59:54.613903 8925 raft_consensus.cc:686] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771 [term 1 LEADER]: Becoming Leader. State: Replica: 134ca7183ef7455691a15a71077ce771, State: Running, Role: LEADER
I20260812 00:59:54.614457 8925 consensus_queue.cc:237] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ce59fd38574c69bf5147c41ae8cfa0" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 46295 } } peers { permanent_uuid: "134ca7183ef7455691a15a71077ce771" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37571 } } peers { permanent_uuid: "ddbe3f402cec4142b962542377361bc4" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33645 } }
I20260812 00:59:54.621156 7977 catalog_manager.cc:5495] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771 reported cstate change: term changed from 0 to 1, leader changed from <none> to 134ca7183ef7455691a15a71077ce771 (127.21.201.67). New cstate: current_term: 1 leader_uuid: "134ca7183ef7455691a15a71077ce771" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ce59fd38574c69bf5147c41ae8cfa0" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 46295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "134ca7183ef7455691a15a71077ce771" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddbe3f402cec4142b962542377361bc4" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33645 } health_report { overall_health: UNKNOWN } } }
I20260812 00:59:54.708926 8905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:54.703113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41464 (local address 127.21.201.67:37571)
0812 00:59:54.703382 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:54.703387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:54.703408 (+ 21us) server_negotiation.cc:407] Connection header received
0812 00:59:54.703590 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:54.703594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:54.703682 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:54.703878 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:59:54.704277 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.705015 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.706541 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.707000 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.707102 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:54.707656 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:59:54.708289 (+ 633us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:59:54.708414 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:54.708518 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 00:59:54.708593 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260812 00:59:54.719422 8271 raft_consensus.cc:1261] T 97d4b54eb8be4462aae2732709413844 P 69ce59fd38574c69bf5147c41ae8cfa0 [term 1 FOLLOWER]: Refusing update from remote peer 134ca7183ef7455691a15a71077ce771: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:59:54.719476 8537 raft_consensus.cc:1261] T 97d4b54eb8be4462aae2732709413844 P ddbe3f402cec4142b962542377361bc4 [term 1 FOLLOWER]: Refusing update from remote peer 134ca7183ef7455691a15a71077ce771: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:59:54.720584 8928 consensus_queue.cc:1035] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddbe3f402cec4142b962542377361bc4" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:59:54.721006 8925 consensus_queue.cc:1035] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ce59fd38574c69bf5147c41ae8cfa0" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 46295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:59:54.752254 7978 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:59:54.752584 7978 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:59:54.756687 7978 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7624bb4df12f43f78a18b3934487d0ca: Sending DeleteTablet for 3 replicas of tablet 97d4b54eb8be4462aae2732709413844
I20260812 00:59:54.758119 8223 tablet_service.cc:1514] Processing DeleteTablet for tablet 97d4b54eb8be4462aae2732709413844 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:59:54 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:33148
I20260812 00:59:54.758527 8501 tablet_service.cc:1514] Processing DeleteTablet for tablet 97d4b54eb8be4462aae2732709413844 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:59:54 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:59178
I20260812 00:59:54.758921 8745 tablet_service.cc:1514] Processing DeleteTablet for tablet 97d4b54eb8be4462aae2732709413844 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:59:54 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:41452
I20260812 00:59:54.759626 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 8103
I20260812 00:59:54.759752 8955 tablet_replica.cc:330] T 97d4b54eb8be4462aae2732709413844 P ddbe3f402cec4142b962542377361bc4: stopping tablet replica
I20260812 00:59:54.760303 8955 raft_consensus.cc:2229] T 97d4b54eb8be4462aae2732709413844 P ddbe3f402cec4142b962542377361bc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:59:54.760869 8955 raft_consensus.cc:2258] T 97d4b54eb8be4462aae2732709413844 P ddbe3f402cec4142b962542377361bc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:59:54.762730 8954 tablet_replica.cc:330] T 97d4b54eb8be4462aae2732709413844 P 69ce59fd38574c69bf5147c41ae8cfa0: stopping tablet replica
I20260812 00:59:54.763187 8954 raft_consensus.cc:2229] T 97d4b54eb8be4462aae2732709413844 P 69ce59fd38574c69bf5147c41ae8cfa0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:59:54.763862 8954 raft_consensus.cc:2258] T 97d4b54eb8be4462aae2732709413844 P 69ce59fd38574c69bf5147c41ae8cfa0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:59:54.764194 8955 ts_tablet_manager.cc:1905] T 97d4b54eb8be4462aae2732709413844 P ddbe3f402cec4142b962542377361bc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:59:54.765478 8957 tablet_replica.cc:330] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771: stopping tablet replica
I20260812 00:59:54.766008 8957 raft_consensus.cc:2229] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:59:54.766974 8957 raft_consensus.cc:2258] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:59:54.769344 8954 ts_tablet_manager.cc:1905] T 97d4b54eb8be4462aae2732709413844 P 69ce59fd38574c69bf5147c41ae8cfa0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:59:54.773268 8955 ts_tablet_manager.cc:1918] T 97d4b54eb8be4462aae2732709413844 P ddbe3f402cec4142b962542377361bc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:59:54.775547 8957 ts_tablet_manager.cc:1905] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 12 00:59:54 dist-test-slave-7f01 krb5kdc[7900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.65@KRBTEST.COM
I20260812 00:59:54.778482 8954 ts_tablet_manager.cc:1918] T 97d4b54eb8be4462aae2732709413844 P 69ce59fd38574c69bf5147c41ae8cfa0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:59:54.778805 8954 log.cc:1196] T 97d4b54eb8be4462aae2732709413844 P 69ce59fd38574c69bf5147c41ae8cfa0: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-0/wal/wals/97d4b54eb8be4462aae2732709413844
I20260812 00:59:54.779598 8954 ts_tablet_manager.cc:1939] T 97d4b54eb8be4462aae2732709413844 P 69ce59fd38574c69bf5147c41ae8cfa0: Deleting consensus metadata
I20260812 00:59:54.779933 8955 log.cc:1196] T 97d4b54eb8be4462aae2732709413844 P ddbe3f402cec4142b962542377361bc4: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/97d4b54eb8be4462aae2732709413844
I20260812 00:59:54.780663 8955 ts_tablet_manager.cc:1939] T 97d4b54eb8be4462aae2732709413844 P ddbe3f402cec4142b962542377361bc4: Deleting consensus metadata
I20260812 00:59:54.781944 7957 catalog_manager.cc:4841] TS 69ce59fd38574c69bf5147c41ae8cfa0 (127.21.201.65:46295): tablet 97d4b54eb8be4462aae2732709413844 (table test-table [id=bca3c332820d487fafc99593321b1ca9]) successfully deleted
I20260812 00:59:54.784412 7957 catalog_manager.cc:4841] TS ddbe3f402cec4142b962542377361bc4 (127.21.201.66:33645): tablet 97d4b54eb8be4462aae2732709413844 (table test-table [id=bca3c332820d487fafc99593321b1ca9]) successfully deleted
I20260812 00:59:54.785844 8903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:54.762867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33164 (local address 127.21.201.65:46295)
0812 00:59:54.763057 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:54.763066 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:54.764753 (+ 1687us) server_negotiation.cc:407] Connection header received
0812 00:59:54.764964 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:54.764972 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:54.765096 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:54.765210 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:54.767147 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.768342 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.770636 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.771458 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.771567 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:54.777432 (+ 5865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:54.777448 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:54.777466 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:54.777515 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:54.781917 (+ 4402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:54.782950 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:54.782964 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:54.782967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:54.783111 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:54.783548 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:54.783553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:54.783556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:54.784185 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:54.784506 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:54.785063 (+ 557us) server_negotiation.cc:299] Negotiation successful
0812 00:59:54.785297 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":61}
I20260812 00:59:54.786580 8348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:54.786810 8957 ts_tablet_manager.cc:1918] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:59:54.787107 8957 log.cc:1196] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/97d4b54eb8be4462aae2732709413844
I20260812 00:59:54.787958 8957 ts_tablet_manager.cc:1939] T 97d4b54eb8be4462aae2732709413844 P 134ca7183ef7455691a15a71077ce771: Deleting consensus metadata
I20260812 00:59:54.790634 7959 catalog_manager.cc:4841] TS 134ca7183ef7455691a15a71077ce771 (127.21.201.67:37571): tablet 97d4b54eb8be4462aae2732709413844 (table test-table [id=bca3c332820d487fafc99593321b1ca9]) successfully deleted
I20260812 00:59:54.898991 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 8103
I20260812 00:59:54.933018 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 8376
Aug 12 00:59:54 dist-test-slave-7f01 krb5kdc[7900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.66@KRBTEST.COM
I20260812 00:59:54.949011 8907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:54.933886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59190 (local address 127.21.201.66:33645)
0812 00:59:54.934114 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:54.934121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:54.934142 (+ 21us) server_negotiation.cc:407] Connection header received
0812 00:59:54.934332 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:54.934342 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:54.934412 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:54.934525 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:54.935786 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.936574 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.937877 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:54.938297 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:54.938446 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:54.942803 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:54.942821 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:54.942838 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:54.942884 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:54.945835 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:54.946605 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:54.946618 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:54.946623 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:54.946842 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:54.947216 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:54.947222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:54.947225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:54.947788 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:54.948080 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:54.948359 (+ 279us) server_negotiation.cc:299] Negotiation successful
0812 00:59:54.948606 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 00:59:54.949697 8617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:55.043013 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 8376
I20260812 00:59:55.071674 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 8634
Aug 12 00:59:55 dist-test-slave-7f01 krb5kdc[7900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.67@KRBTEST.COM
I20260812 00:59:55.091483 8905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:59:55.072570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41474 (local address 127.21.201.67:37571)
0812 00:59:55.072807 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 00:59:55.072814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:59:55.072838 (+ 24us) server_negotiation.cc:407] Connection header received
0812 00:59:55.072921 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:59:55.072927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:59:55.073071 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:59:55.073205 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:59:55.074503 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:55.075685 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:55.077394 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:59:55.078149 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:59:55.078312 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:59:55.083647 (+ 5335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:59:55.083667 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:59:55.083690 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:59:55.083815 (+ 125us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:59:55.088299 (+ 4484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:55.089087 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:55.089096 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:55.089100 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:55.089291 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:59:55.089675 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:59:55.089682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:59:55.089690 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:59:55.090215 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:59:55.090561 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0812 00:59:55.090779 (+ 218us) server_negotiation.cc:299] Negotiation successful
0812 00:59:55.091034 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 00:59:55.092406 8871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:55.223369 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 8634
I20260812 00:59:55.260250 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 7920
I20260812 00:59:55.261584 8039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 00:59:55.388118 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 7920
2026-08-12T00:59:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (5122 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:59:55 dist-test-slave-7f01 krb5kdc[8969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:59:55 dist-test-slave-7f01 krb5kdc[8969](info): set up 2 sockets
Aug 12 00:59:55 dist-test-slave-7f01 krb5kdc[8969](info): commencing operation
krb5kdc: starting...
W20260812 00:59:56.801430 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.363s user 0.001s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 00:59:56 dist-test-slave-7f01 krb5kdc[8969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:59:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:59:57Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.201.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.201.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.201.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/rangeradmin_127.21.201.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.201.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/rangeradmin_127.21.201.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.201.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.201.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.201.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/rangerlookup_127.21.201.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.201.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/rangerlookup_127.21.201.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.201.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.201.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.201.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/HTTP_127.21.201.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.201.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/HTTP_127.21.201.84@KRBTEST.COM.keytab.
I20260812 00:59:57.235446 22309 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskJMOGj2/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/postgres -l logfile start
2026-08-12 00:59:59.883 UTC [9034] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 00:59:59.883 UTC [9034] LOG: listening on IPv4 address "127.21.201.84", port 49603
2026-08-12 00:59:59.886 UTC [9034] LOG: listening on Unix socket "/tmp/.s.PGSQL.49603"
2026-08-12 00:59:59.912 UTC [9037] LOG: database system was shut down at 2026-08-12 00:59:58 UTC
2026-08-12 00:59:59.917 UTC [9034] LOG: database system is ready to accept connections
I20260812 01:00:01.158005 22309 mini_postgres.cc:96] Postgres bound to 49603
2026-08-12 01:00:01.183 UTC [9050] FATAL: database "slave" does not exist
127.21.201.84:49603 - accepting connections
I20260812 01:00:01.184083 22309 mini_ranger.cc:161] Starting Ranger...
I20260812 01:00:01.218689 22309 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 01:00:01.730878 22309 mini_ranger.cc:87] Created ranger Postgres database
I20260812 01:00:01.731025 22309 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ranger-admin
2026-08-12 01:00:02,056 [I] DB FLAVOR :POSTGRES
2026-08-12 01:00:02,057 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 01:00:02,057 [I] Checking connection..
2026-08-12 01:00:02,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:49603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 01:00:02,331 [I] Checking connection passed.
2026-08-12 01:00:02,331 [I] --------- Verifying version history table ---------
2026-08-12 01:00:02,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:49603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 01:00:02,603 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 01:00:02,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:49603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 01:00:02,851 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 01:00:02,852 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 01:00:02,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:49603/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 01:00:03.117 UTC [9174] WARNING: there is no transaction in progress
2026-08-12 01:00:03,127 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 01:00:03,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:49603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 01:00:03,393 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 01:00:03,393 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 01:00:03,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:49603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 01:00:03,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:49603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 01:00:03,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:49603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7f01.c.gcp-upstream.internal','N') ;"
2026-08-12 01:00:04,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:49603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 01:00:04,458 [I] Table x_portal_user does not exist in database ranger
2026-08-12 01:00:04,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:49603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 01:00:04,736 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 01:00:04,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:49603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 01:00:04,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:49603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 01:00:05,267 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 01:00:05,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:49603/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 01:00:06.286 UTC [9372] WARNING: there is no transaction in progress
2026-08-12 01:00:06.306 UTC [9372] WARNING: there is no transaction in progress
2026-08-12 01:00:06.324 UTC [9372] WARNING: there is no transaction in progress
2026-08-12 01:00:06.341 UTC [9372] WARNING: there is no transaction in progress
2026-08-12 01:00:06.362 UTC [9372] WARNING: there is no transaction in progress
2026-08-12 01:00:06.380 UTC [9372] WARNING: there is no transaction in progress
2026-08-12 01:00:06.452 UTC [9372] WARNING: there is no transaction in progress
2026-08-12 01:00:06.460 UTC [9372] WARNING: there is no transaction in progress
2026-08-12 01:00:06.476 UTC [9372] WARNING: there is no transaction in progress
2026-08-12 01:00:06.486 UTC [9372] WARNING: there is no transaction in progress
2026-08-12 01:00:06.497 UTC [9372] WARNING: there is no transaction in progress
2026-08-12 01:00:06.512 UTC [9372] WARNING: there is no transaction in progress
2026-08-12 01:00:06.523 UTC [9372] WARNING: there is no transaction in progress
2026-08-12 01:00:06.530 UTC [9372] WARNING: there is no transaction in progress
2026-08-12 01:00:06.538 UTC [9372] WARNING: there is no transaction in progress
2026-08-12 01:00:06.545 UTC [9372] WARNING: there is no transaction in progress
2026-08-12 01:00:07,218 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 01:00:07,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:49603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 01:00:07,455 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 01:00:07,455 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:49603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 01:00:07,704 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 01:00:07,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:49603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 01:00:07,961 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 01:00:07,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:49603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 01:00:08,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:49603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 01:00:08,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:49603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7f01.c.gcp-upstream.internal';"
2026-08-12 01:00:08,759 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 01:00:08,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:49603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 01:00:09,039 [I] DB_PATCHES have already been applied
I20260812 01:00:09.045732 22309 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ranger-admin:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskJMOGj2/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 01:00:09.045835 22309 mini_ranger.cc:193] Using host: 127.21.201.84
I20260812 01:00:09.068367 22309 mini_ranger.cc:227] Ranger admin URL: http://127.21.201.84:45241
Aug 12, 2026 1:00:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 1:00:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 1:00:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 1:00:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 1:00:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 1:00:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 1:00:09 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 01:00:10.563676 22309 mini_ranger.cc:160] Time spent starting Ranger: real 9.380s user 0.001s sys 0.068s
Aug 12, 2026 1:00:31 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 1:00:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 12, 2026 1:00:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 1:00:31 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 1:00:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 1:00:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 1:00:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 1:00:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 1:00:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 1:00:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 1:00:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 1:00:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 1:00:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 1:00:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 1:00:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 1:00:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 1:00:31 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 1:00:32 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 01:00:32.946823 22309 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.201.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.201.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.201.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/rangerkms_127.21.201.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.201.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/rangerkms_127.21.201.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.201.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.201.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.201.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/HTTP_127.21.201.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.201.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/HTTP_127.21.201.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 01:00:33 dist-test-slave-7f01 krb5kdc[8969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496433, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 01:00:33.166600 22309 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 01:00:33.195991 22309 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 01:00:34.068747 22309 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 01:00:34.068938 22309 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ranger-kms
I20260812 01:00:34.069082 22309 mini_ranger_kms.cc:227] Using postgres at 127.21.201.84:49603
2026-08-12 01:00:34,629 [I] DB FLAVOR :POSTGRES
2026-08-12 01:00:34,629 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 01:00:34,629 [I] Checking connection
2026-08-12 01:00:34,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:49603/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 01:00:34,871 [I] connection success
2026-08-12 01:00:34,872 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 01:00:34,872 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 01:00:34,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:49603/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 01:00:35,139 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 01:00:35,139 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 01:00:35,139 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 01:00:35,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:49603/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 01:00:35,421 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 01:00:35.926780 22309 mini_ranger_kms.cc:310] Created kms service
I20260812 01:00:36.139986 22309 mini_ranger_kms.cc:326] Created kudu user
I20260812 01:00:36.191035 22309 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 01:00:36 dist-test-slave-7f01 krb5kdc[8969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496433, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.201.84@KRBTEST.COM
I20260812 01:00:36.640187 22309 mini_ranger_kms.cc:384] Added ranger policy
I20260812 01:00:36.641662 22309 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ranger-kms:/tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 01:00:36.641729 22309 mini_ranger_kms.cc:240] Using host: 127.21.201.84
I20260812 01:00:36.669112 22309 mini_ranger_kms.cc:276] Ranger KMS PID: 9709
I20260812 01:00:36.669358 22309 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.201.84:59679
Aug 12, 2026 1:00:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 1:00:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 1:00:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 1:00:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 1:00:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 1:00:38 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 01:00:39 dist-test-slave-7f01 krb5kdc[8969](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786496439, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.201.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 01:00:39 dist-test-slave-7f01 krb5kdc[8969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496439, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.201.84@KRBTEST.COM for HTTP/127.21.201.84@KRBTEST.COM
I20260812 01:00:39.586861 22309 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.420s user 0.136s sys 0.469s
I20260812 01:00:39.587226 22309 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 01:00:39.587479 22309 mini_ranger_kms.cc:401] 127.21.201.84:59679/kms/v1/keys
Aug 12 01:00:39 dist-test-slave-7f01 krb5kdc[8969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496439, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.201.84@KRBTEST.COM for HTTP/127.21.201.84@KRBTEST.COM
Aug 12 01:00:42 dist-test-slave-7f01 krb5kdc[8969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 01:00:42.403216 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:34079
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:34079
--ranger_config_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.201.84:59679/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:00:42.800367 9795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:00:42.800693 9795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:00:42.800863 9795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:00:42.808998 9795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:00:42.809139 9795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:00:42.809275 9795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:00:42.809372 9795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:00:42.809454 9795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:00:42.820617 9795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.201.84:59679/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:34079
--ranger_config_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:34079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:00:42.822093 9795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:00:42.823495 9795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:00:42.836843 9806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:00:42.836808 9812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:00:42.836762 9810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:00:42.837813 9795 server_base.cc:1034] running on GCE node
I20260812 01:00:42.838678 9795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:00:42.840287 9795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:00:42.841544 9795 hybrid_clock.cc:648] HybridClock initialized: now 1786496442841522 us; error 76 us; skew 500 ppm
I20260812 01:00:42.841922 9795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:00:42 dist-test-slave-7f01 krb5kdc[8969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496442, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:00:42.848044 9795 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 01:00:42.849712 9795 webserver.cc:460] Webserver started at http://127.21.201.126:42763/ using document root <none> and password file <none>
I20260812 01:00:42.850397 9795 fs_manager.cc:359] Metadata directory not provided
I20260812 01:00:42.850541 9795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:00:42.850872 9795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 01:00:42 dist-test-slave-7f01 krb5kdc[8969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496442, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for HTTP/127.21.201.84@KRBTEST.COM
I20260812 01:00:42.908139 9795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "7cc48f8c8c92431aaa088f6eb0022bb4"
format_stamp: "Formatted at 2026-08-12 01:00:42 on dist-test-slave-7f01"
server_key: "b582de8b2933714579355a77a0690aff"
server_key_iv: "a6d17fe07e4b5885d267a744f2133aba"
server_key_version: "kuduclusterkey@0"
I20260812 01:00:42.909308 9795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "7cc48f8c8c92431aaa088f6eb0022bb4"
format_stamp: "Formatted at 2026-08-12 01:00:42 on dist-test-slave-7f01"
server_key: "b582de8b2933714579355a77a0690aff"
server_key_iv: "a6d17fe07e4b5885d267a744f2133aba"
server_key_version: "kuduclusterkey@0"
I20260812 01:00:42.917223 9795 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260812 01:00:42.939353 9828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:00:42.941249 9795 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 01:00:42.941469 9795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "7cc48f8c8c92431aaa088f6eb0022bb4"
format_stamp: "Formatted at 2026-08-12 01:00:42 on dist-test-slave-7f01"
server_key: "b582de8b2933714579355a77a0690aff"
server_key_iv: "a6d17fe07e4b5885d267a744f2133aba"
server_key_version: "kuduclusterkey@0"
I20260812 01:00:42.941746 9795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:00:42.973834 9795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:00:42.979331 9795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:00:42.979756 9795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:00:43.002592 9795 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:34079
I20260812 01:00:43.002651 9908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:34079 every 8 connection(s)
I20260812 01:00:43.004387 9795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:00:43.008778 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:00:43.009032 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 9795
I20260812 01:00:43.009263 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:00:43.010140 22309 external_mini_cluster.cc:1428] Setting key 9fa8f4a103195b6f531f705d8a4320d5
I20260812 01:00:43.028447 9909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cc48f8c8c92431aaa088f6eb0022bb4: Bootstrap starting.
I20260812 01:00:43.032770 9909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cc48f8c8c92431aaa088f6eb0022bb4: Neither blocks nor log segments found. Creating new log.
I20260812 01:00:43.034104 9909 log.cc:824] T 00000000000000000000000000000000 P 7cc48f8c8c92431aaa088f6eb0022bb4: Log is configured to *not* fsync() on all Append() calls
I20260812 01:00:43.037585 9909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cc48f8c8c92431aaa088f6eb0022bb4: No bootstrap required, opened a new log
Aug 12 01:00:43 dist-test-slave-7f01 krb5kdc[8969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:00:43.048991 9909 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cc48f8c8c92431aaa088f6eb0022bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cc48f8c8c92431aaa088f6eb0022bb4" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 34079 } }
I20260812 01:00:43.049463 9909 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cc48f8c8c92431aaa088f6eb0022bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:00:43.049571 9909 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cc48f8c8c92431aaa088f6eb0022bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cc48f8c8c92431aaa088f6eb0022bb4, State: Initialized, Role: FOLLOWER
I20260812 01:00:43.050225 9916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:00:43.012483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36420 (local address 127.21.201.126:34079)
0812 01:00:43.027472 (+ 14989us) server_negotiation.cc:206] Beginning negotiation
0812 01:00:43.027489 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 01:00:43.027590 (+ 101us) server_negotiation.cc:407] Connection header received
0812 01:00:43.030194 (+ 2604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:00:43.030230 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:00:43.030857 (+ 627us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:00:43.031338 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:00:43.033064 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:43.035242 (+ 2178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:43.036682 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:43.037385 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:43.037575 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:00:43.041735 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:00:43.041747 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:00:43.041767 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:00:43.041827 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:00:43.045249 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:00:43.046040 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:00:43.046047 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:00:43.046054 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:00:43.046292 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:00:43.046729 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:00:43.046733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:00:43.046735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:00:43.047485 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:00:43.047957 (+ 472us) server_negotiation.cc:1035] Waiting for connection context
0812 01:00:43.048445 (+ 488us) server_negotiation.cc:299] Negotiation successful
0812 01:00:43.048896 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14441,"thread_start_us":14522,"threads_started":1}
I20260812 01:00:43.050263 9909 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cc48f8c8c92431aaa088f6eb0022bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cc48f8c8c92431aaa088f6eb0022bb4" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 34079 } }
I20260812 01:00:43.050594 9909 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cc48f8c8c92431aaa088f6eb0022bb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:00:43.050985 9909 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cc48f8c8c92431aaa088f6eb0022bb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:00:43.051261 9909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cc48f8c8c92431aaa088f6eb0022bb4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:00:43.055164 9909 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cc48f8c8c92431aaa088f6eb0022bb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cc48f8c8c92431aaa088f6eb0022bb4" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 34079 } }
I20260812 01:00:43.055768 9909 leader_election.cc:302] T 00000000000000000000000000000000 P 7cc48f8c8c92431aaa088f6eb0022bb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cc48f8c8c92431aaa088f6eb0022bb4; no voters:
I20260812 01:00:43.056418 9909 leader_election.cc:288] T 00000000000000000000000000000000 P 7cc48f8c8c92431aaa088f6eb0022bb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:00:43.056551 9919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cc48f8c8c92431aaa088f6eb0022bb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:00:43.058723 9909 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cc48f8c8c92431aaa088f6eb0022bb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:00:43.062564 9919 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cc48f8c8c92431aaa088f6eb0022bb4 [term 1 LEADER]: Becoming Leader. State: Replica: 7cc48f8c8c92431aaa088f6eb0022bb4, State: Running, Role: LEADER
I20260812 01:00:43.063345 9919 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cc48f8c8c92431aaa088f6eb0022bb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cc48f8c8c92431aaa088f6eb0022bb4" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 34079 } }
I20260812 01:00:43.067014 9923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cc48f8c8c92431aaa088f6eb0022bb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cc48f8c8c92431aaa088f6eb0022bb4. Latest consensus state: current_term: 1 leader_uuid: "7cc48f8c8c92431aaa088f6eb0022bb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cc48f8c8c92431aaa088f6eb0022bb4" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 34079 } } }
I20260812 01:00:43.067414 9923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cc48f8c8c92431aaa088f6eb0022bb4 [sys.catalog]: This master's current role is: LEADER
I20260812 01:00:43.068091 9922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cc48f8c8c92431aaa088f6eb0022bb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cc48f8c8c92431aaa088f6eb0022bb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cc48f8c8c92431aaa088f6eb0022bb4" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 34079 } } }
I20260812 01:00:43.068471 9922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cc48f8c8c92431aaa088f6eb0022bb4 [sys.catalog]: This master's current role is: LEADER
I20260812 01:00:43.078714 9942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:00:43.085454 9942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:00:43.087342 9909 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 01:00:43.117611 9942 catalog_manager.cc:1347] Generated new cluster ID: 6a29a1746224438194b41bf041302662
I20260812 01:00:43.117837 9942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:00:43.153600 9942 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:00:43.155902 9942 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:00:43.163807 9942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cc48f8c8c92431aaa088f6eb0022bb4: Generated new TSK 0
I20260812 01:00:43.164884 9942 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.201.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
I20260812 01:00:43.980049 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.201.84:59679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:0
--local_ip_for_outbound_sockets=127.21.201.65
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:34079
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 01:00:44 dist-test-slave-7f01 krb5kdc[8969](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786496444, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 01:00:44.442109 10006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:00:44.442567 10006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:00:44.442790 10006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:00:44.455302 10006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:00:44.455549 10006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:00:44.455812 10006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 01:00:44.471056 10006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.201.84:59679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:34079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:00:44.472993 10006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:00:44.474990 10006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:00:44.496927 10025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:00:44.497092 10029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:00:44.505436 10006 server_base.cc:1034] running on GCE node
W20260812 01:00:44.505714 10035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:00:44.506573 10006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:00:44.507947 10006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:00:44.509263 10006 hybrid_clock.cc:648] HybridClock initialized: now 1786496444509190 us; error 60 us; skew 500 ppm
I20260812 01:00:44.509892 10006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:00:44 dist-test-slave-7f01 krb5kdc[8969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496444, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:00:44.514981 10006 init.cc:377] Logged in from keytab as kudu/127.21.201.65@KRBTEST.COM (short username kudu)
I20260812 01:00:44.517381 10006 webserver.cc:460] Webserver started at http://127.21.201.65:46519/ using document root <none> and password file <none>
I20260812 01:00:44.518518 10006 fs_manager.cc:359] Metadata directory not provided
I20260812 01:00:44.518743 10006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:00:44.519290 10006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 01:00:44 dist-test-slave-7f01 krb5kdc[8969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496444, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for HTTP/127.21.201.84@KRBTEST.COM
I20260812 01:00:44.544107 10006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-0/data/instance:
uuid: "5bee060365b94dc5a752816a2e7ca6ac"
format_stamp: "Formatted at 2026-08-12 01:00:44 on dist-test-slave-7f01"
server_key: "ad87392609084d398e725ff0c170c153"
server_key_iv: "6cef7e55ac815fde7d8e0c50bd758858"
server_key_version: "kuduclusterkey@0"
I20260812 01:00:44.545632 10006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance:
uuid: "5bee060365b94dc5a752816a2e7ca6ac"
format_stamp: "Formatted at 2026-08-12 01:00:44 on dist-test-slave-7f01"
server_key: "ad87392609084d398e725ff0c170c153"
server_key_iv: "6cef7e55ac815fde7d8e0c50bd758858"
server_key_version: "kuduclusterkey@0"
I20260812 01:00:44.554162 10006 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260812 01:00:44.592448 10044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:00:44.594522 10006 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.007s sys 0.007s
I20260812 01:00:44.594774 10006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "5bee060365b94dc5a752816a2e7ca6ac"
format_stamp: "Formatted at 2026-08-12 01:00:44 on dist-test-slave-7f01"
server_key: "ad87392609084d398e725ff0c170c153"
server_key_iv: "6cef7e55ac815fde7d8e0c50bd758858"
server_key_version: "kuduclusterkey@0"
I20260812 01:00:44.595089 10006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:00:44.610654 10006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:00:44.615291 10006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:00:44.615636 10006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:00:44.616765 10006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:00:44.618302 10006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:00:44.618409 10006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:00:44.618552 10006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:00:44.618664 10006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:00:44.725018 10006 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:34503
I20260812 01:00:44.725067 10273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:34503 every 8 connection(s)
I20260812 01:00:44.727082 10006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 01:00:44.729684 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 10006
I20260812 01:00:44.730088 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance
I20260812 01:00:44.731065 22309 external_mini_cluster.cc:1428] Setting key 87ad130c23226713a45875daeb5aeb79
Aug 12 01:00:44 dist-test-slave-7f01 krb5kdc[8969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496444, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.201.66@KRBTEST.COM; defaulting to no policy
I20260812 01:00:44.785974 10280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:00:44.737408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:39743 (local address 127.21.201.126:34079)
0812 01:00:44.755974 (+ 18566us) server_negotiation.cc:206] Beginning negotiation
0812 01:00:44.755982 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:00:44.756016 (+ 34us) server_negotiation.cc:407] Connection header received
0812 01:00:44.758313 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:00:44.758320 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:00:44.758426 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:00:44.758558 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:00:44.763096 (+ 4538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:44.764470 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:44.767531 (+ 3061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:44.768353 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:44.768477 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:00:44.775952 (+ 7475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:00:44.775962 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:00:44.775971 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:00:44.776059 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:00:44.780766 (+ 4707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:00:44.781995 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:00:44.782002 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:00:44.782007 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:00:44.782276 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:00:44.783030 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:00:44.783036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:00:44.783040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:00:44.783400 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:00:44.783570 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 01:00:44.784651 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0812 01:00:44.785456 (+ 805us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18380,"thread_start_us":18394,"threads_started":1}
I20260812 01:00:44.786679 10278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:00:44.736646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:34079 (local address 127.21.201.65:39743)
0812 01:00:44.750401 (+ 13755us) negotiation.cc:107] Waiting for socket to connect
0812 01:00:44.750481 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0812 01:00:44.757758 (+ 7277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:00:44.760873 (+ 3115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:00:44.760889 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:00:44.761702 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:00:44.762732 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:00:44.762864 (+ 132us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:00:44.764695 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:00:44.764703 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:00:44.767290 (+ 2587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:00:44.767350 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:00:44.768936 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:00:44.768945 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:00:44.769325 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:00:44.770291 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:00:44.770328 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:00:44.775569 (+ 5241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:00:44.780986 (+ 5417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:00:44.780999 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:00:44.781022 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:00:44.781822 (+ 800us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:00:44.782544 (+ 722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:00:44.782548 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:00:44.782550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:00:44.782859 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:00:44.783552 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:00:44.783561 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:00:44.784099 (+ 538us) client_negotiation.cc:769] Sending connection context
0812 01:00:44.784467 (+ 368us) client_negotiation.cc:240] Negotiation successful
0812 01:00:44.785070 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13247,"thread_start_us":14942,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.66@KRBTEST.COM" created.
I20260812 01:00:44.789316 10274 heartbeater.cc:347] Connected to a master server at 127.21.201.126:34079
I20260812 01:00:44.789669 10274 heartbeater.cc:464] Registering TS with master...
I20260812 01:00:44.790617 10274 heartbeater.cc:511] Master 127.21.201.126:34079 requested a full tablet report, sending...
I20260812 01:00:44.793231 9856 ts_manager.cc:194] Registered new tserver with Master: 5bee060365b94dc5a752816a2e7ca6ac (127.21.201.65:34503)
I20260812 01:00:44.794845 9856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.65@KRBTEST.COM'} at 127.21.201.65:39743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
I20260812 01:00:44.827632 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.201.84:59679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:0
--local_ip_for_outbound_sockets=127.21.201.66
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:34079
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 01:00:44 dist-test-slave-7f01 krb5kdc[8969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496444, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for HTTP/127.21.201.84@KRBTEST.COM
Aug 12 01:00:45 dist-test-slave-7f01 krb5kdc[8969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496444, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for HTTP/127.21.201.84@KRBTEST.COM
W20260812 01:00:45.414428 10285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:00:45.414934 10285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:00:45.415148 10285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:00:45.428260 10285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:00:45.428542 10285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:00:45.428769 10285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 01:00:45.445608 10285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.201.84:59679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:34079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:00:45.447906 10285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:00:45.449837 10285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:00:45.493851 10298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:00:45.494870 10296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:00:45.498682 10304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:00:45.500722 10285 server_base.cc:1034] running on GCE node
I20260812 01:00:45.501855 10285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:00:45.503358 10285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:00:45.504668 10285 hybrid_clock.cc:648] HybridClock initialized: now 1786496445504598 us; error 51 us; skew 500 ppm
I20260812 01:00:45.505117 10285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:00:45 dist-test-slave-7f01 krb5kdc[8969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496445, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:00:45.509918 10285 init.cc:377] Logged in from keytab as kudu/127.21.201.66@KRBTEST.COM (short username kudu)
I20260812 01:00:45.514897 10285 webserver.cc:460] Webserver started at http://127.21.201.66:38229/ using document root <none> and password file <none>
I20260812 01:00:45.516078 10285 fs_manager.cc:359] Metadata directory not provided
I20260812 01:00:45.516314 10285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:00:45.516783 10285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 01:00:45 dist-test-slave-7f01 krb5kdc[8969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496445, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for HTTP/127.21.201.84@KRBTEST.COM
I20260812 01:00:45.547416 10285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-1/data/instance:
uuid: "34b33aeb9dd04a70a16cff2b843c1269"
format_stamp: "Formatted at 2026-08-12 01:00:45 on dist-test-slave-7f01"
server_key: "94d11d361ea64400ed877742cc88638b"
server_key_iv: "a18ba946ff0e616082d31579223085c0"
server_key_version: "kuduclusterkey@0"
I20260812 01:00:45.548560 10285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance:
uuid: "34b33aeb9dd04a70a16cff2b843c1269"
format_stamp: "Formatted at 2026-08-12 01:00:45 on dist-test-slave-7f01"
server_key: "94d11d361ea64400ed877742cc88638b"
server_key_iv: "a18ba946ff0e616082d31579223085c0"
server_key_version: "kuduclusterkey@0"
I20260812 01:00:45.558782 10285 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.006s sys 0.002s
I20260812 01:00:45.579633 10329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:00:45.581431 10285 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 01:00:45.581719 10285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "34b33aeb9dd04a70a16cff2b843c1269"
format_stamp: "Formatted at 2026-08-12 01:00:45 on dist-test-slave-7f01"
server_key: "94d11d361ea64400ed877742cc88638b"
server_key_iv: "a18ba946ff0e616082d31579223085c0"
server_key_version: "kuduclusterkey@0"
I20260812 01:00:45.582058 10285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:00:45.617681 10285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:00:45.622226 10285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:00:45.622581 10285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:00:45.623800 10285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:00:45.625311 10285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:00:45.625418 10285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:00:45.625576 10285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:00:45.625696 10285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:00:45.684965 10285 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:43573
I20260812 01:00:45.685109 10514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:43573 every 8 connection(s)
I20260812 01:00:45.686697 10285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 01:00:45.694465 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 10285
I20260812 01:00:45.694669 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance
I20260812 01:00:45.695317 22309 external_mini_cluster.cc:1428] Setting key befb371c348c6e2ac7ad5d68e6a249a1
Aug 12 01:00:45 dist-test-slave-7f01 krb5kdc[8969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496445, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:00:45.714457 10520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:00:45.689933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:35687 (local address 127.21.201.126:34079)
0812 01:00:45.690656 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0812 01:00:45.690661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:00:45.691944 (+ 1283us) server_negotiation.cc:407] Connection header received
0812 01:00:45.694215 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:00:45.694221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:00:45.694325 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:00:45.694466 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:00:45.696842 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:45.697920 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:45.700541 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:45.701283 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:45.701432 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:00:45.706775 (+ 5343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:00:45.706782 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:00:45.706788 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:00:45.706830 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:00:45.710629 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:00:45.711648 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:00:45.711653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:00:45.711655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:00:45.711854 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:00:45.712419 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:00:45.712424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:00:45.712428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:00:45.712752 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:00:45.712939 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0812 01:00:45.713771 (+ 832us) server_negotiation.cc:299] Negotiation successful
0812 01:00:45.714044 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":494,"threads_started":1}
I20260812 01:00:45.715817 10521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:00:45.690675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:34079 (local address 127.21.201.66:35687)
0812 01:00:45.691722 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0812 01:00:45.691816 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0812 01:00:45.693875 (+ 2059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:00:45.694785 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:00:45.694797 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:00:45.695560 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:00:45.696653 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:00:45.696680 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:00:45.698090 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:00:45.698096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:00:45.700292 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:00:45.700318 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:00:45.701549 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:00:45.701556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:00:45.701822 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:00:45.702687 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:00:45.702723 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:00:45.706389 (+ 3666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:00:45.710811 (+ 4422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:00:45.710823 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:00:45.710845 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:00:45.711497 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:00:45.711969 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:00:45.711974 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:00:45.711978 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:00:45.712286 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:00:45.712894 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:00:45.712902 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:00:45.713337 (+ 435us) client_negotiation.cc:769] Sending connection context
0812 01:00:45.713667 (+ 330us) client_negotiation.cc:240] Negotiation successful
0812 01:00:45.714287 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":540,"threads_started":1}
I20260812 01:00:45.717437 10515 heartbeater.cc:347] Connected to a master server at 127.21.201.126:34079
I20260812 01:00:45.717775 10515 heartbeater.cc:464] Registering TS with master...
I20260812 01:00:45.718539 10515 heartbeater.cc:511] Master 127.21.201.126:34079 requested a full tablet report, sending...
I20260812 01:00:45.720119 9856 ts_manager.cc:194] Registered new tserver with Master: 34b33aeb9dd04a70a16cff2b843c1269 (127.21.201.66:43573)
I20260812 01:00:45.720995 9856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.66@KRBTEST.COM'} at 127.21.201.66:35687
WARNING: no policy specified for kudu/127.21.201.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
I20260812 01:00:45.773269 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.201.84:59679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:0
--local_ip_for_outbound_sockets=127.21.201.67
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:34079
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:00:45.799213 10274 heartbeater.cc:503] Master 127.21.201.126:34079 was elected leader, sending a full tablet report...
W20260812 01:00:46.229424 10524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:00:46.229740 10524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:00:46.229863 10524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:00:46.237298 10524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:00:46.237413 10524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:00:46.237537 10524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 01:00:46.248148 10524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.201.84:59679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:34079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:00:46.249506 10524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:00:46.250823 10524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:00:46.265280 10543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:00:46.265378 10539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:00:46.265372 10541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:00:46.266517 10524 server_base.cc:1034] running on GCE node
I20260812 01:00:46.267155 10524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:00:46.268177 10524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:00:46.269405 10524 hybrid_clock.cc:648] HybridClock initialized: now 1786496446269358 us; error 53 us; skew 500 ppm
I20260812 01:00:46.269726 10524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:00:46 dist-test-slave-7f01 krb5kdc[8969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496446, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:00:46.273229 10524 init.cc:377] Logged in from keytab as kudu/127.21.201.67@KRBTEST.COM (short username kudu)
I20260812 01:00:46.274786 10524 webserver.cc:460] Webserver started at http://127.21.201.67:35241/ using document root <none> and password file <none>
I20260812 01:00:46.276024 10524 fs_manager.cc:359] Metadata directory not provided
I20260812 01:00:46.276178 10524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:00:46.276614 10524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 01:00:46 dist-test-slave-7f01 krb5kdc[8969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496446, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for HTTP/127.21.201.84@KRBTEST.COM
I20260812 01:00:46.294220 10524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-2/data/instance:
uuid: "45bb05912e7c4882b6afc3ec126b7cbb"
format_stamp: "Formatted at 2026-08-12 01:00:46 on dist-test-slave-7f01"
server_key: "93cb5c627a85a6cdd7c2f3f95f10e40b"
server_key_iv: "61f4e08d5054437361531bc7b588c966"
server_key_version: "kuduclusterkey@0"
I20260812 01:00:46.295135 10524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance:
uuid: "45bb05912e7c4882b6afc3ec126b7cbb"
format_stamp: "Formatted at 2026-08-12 01:00:46 on dist-test-slave-7f01"
server_key: "93cb5c627a85a6cdd7c2f3f95f10e40b"
server_key_iv: "61f4e08d5054437361531bc7b588c966"
server_key_version: "kuduclusterkey@0"
I20260812 01:00:46.301179 10524 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 01:00:46.317746 10555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:00:46.319168 10524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 01:00:46.319362 10524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "45bb05912e7c4882b6afc3ec126b7cbb"
format_stamp: "Formatted at 2026-08-12 01:00:46 on dist-test-slave-7f01"
server_key: "93cb5c627a85a6cdd7c2f3f95f10e40b"
server_key_iv: "61f4e08d5054437361531bc7b588c966"
server_key_version: "kuduclusterkey@0"
I20260812 01:00:46.319600 10524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:00:46.336006 10524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:00:46.340440 10524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:00:46.340829 10524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:00:46.341931 10524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:00:46.343451 10524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:00:46.343559 10524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:00:46.343709 10524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:00:46.343801 10524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260812 01:00:46.391558 10524 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:37279
I20260812 01:00:46.391604 10751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:37279 every 8 connection(s)
I20260812 01:00:46.393024 10524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 01:00:46.401015 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 10524
I20260812 01:00:46.401221 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance
I20260812 01:00:46.401989 22309 external_mini_cluster.cc:1428] Setting key b9e1764850af8ce7fde8d9d3753ace21
Aug 12 01:00:46 dist-test-slave-7f01 krb5kdc[8969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496446, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:00:46.418293 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:00:46.395823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:38205 (local address 127.21.201.126:34079)
0812 01:00:46.398824 (+ 3001us) server_negotiation.cc:206] Beginning negotiation
0812 01:00:46.398832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:00:46.399918 (+ 1086us) server_negotiation.cc:407] Connection header received
0812 01:00:46.402909 (+ 2991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:00:46.402918 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:00:46.403009 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:00:46.403129 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:00:46.405062 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.405936 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.407499 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.408031 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.408144 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:00:46.412002 (+ 3858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:00:46.412007 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:00:46.412010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:00:46.412043 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:00:46.414723 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:00:46.415410 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:00:46.415414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:00:46.415416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:00:46.415567 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:00:46.416227 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:00:46.416238 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:00:46.416241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:00:46.416664 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:00:46.416816 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 01:00:46.417740 (+ 924us) server_negotiation.cc:299] Negotiation successful
0812 01:00:46.417995 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2861,"thread_start_us":2684,"threads_started":1}
I20260812 01:00:46.419235 10757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:00:46.396281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:34079 (local address 127.21.201.67:38205)
0812 01:00:46.399690 (+ 3409us) negotiation.cc:107] Waiting for socket to connect
0812 01:00:46.399777 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0812 01:00:46.402523 (+ 2746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:00:46.403401 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:00:46.403424 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:00:46.404088 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:00:46.404873 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:00:46.404890 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.406082 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.406086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:00:46.407348 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:00:46.407361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.408255 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.408260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:00:46.408465 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:00:46.409186 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:00:46.409207 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:00:46.411684 (+ 2477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:00:46.414860 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:00:46.414867 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:00:46.414879 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:00:46.415314 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:00:46.415713 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:00:46.415717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:00:46.415725 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:00:46.416082 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:00:46.416850 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:00:46.416858 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:00:46.417315 (+ 457us) client_negotiation.cc:769] Sending connection context
0812 01:00:46.417605 (+ 290us) client_negotiation.cc:240] Negotiation successful
0812 01:00:46.418136 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3119,"thread_start_us":2906,"threads_started":1}
I20260812 01:00:46.420607 10752 heartbeater.cc:347] Connected to a master server at 127.21.201.126:34079
I20260812 01:00:46.420917 10752 heartbeater.cc:464] Registering TS with master...
I20260812 01:00:46.421732 10752 heartbeater.cc:511] Master 127.21.201.126:34079 requested a full tablet report, sending...
I20260812 01:00:46.423175 9856 ts_manager.cc:194] Registered new tserver with Master: 45bb05912e7c4882b6afc3ec126b7cbb (127.21.201.67:37279)
I20260812 01:00:46.424259 9856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.67@KRBTEST.COM'} at 127.21.201.67:38205
I20260812 01:00:46.430271 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:00:46.449349 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:00:46.435223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36436 (local address 127.21.201.126:34079)
0812 01:00:46.435391 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 01:00:46.435395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:00:46.436014 (+ 619us) server_negotiation.cc:407] Connection header received
0812 01:00:46.436208 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:00:46.436214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:00:46.436302 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:00:46.436405 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:00:46.437906 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.438706 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.440543 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.441079 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.441214 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:00:46.443916 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:00:46.443922 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:00:46.443925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:00:46.443956 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:00:46.446651 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:00:46.447427 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:00:46.447434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:00:46.447438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:00:46.447601 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:00:46.448052 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:00:46.448056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:00:46.448058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:00:46.448329 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:00:46.448523 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0812 01:00:46.448879 (+ 356us) server_negotiation.cc:299] Negotiation successful
0812 01:00:46.449057 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 01:00:46.453960 9856 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:00:46.462374 9856 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:00:46.493691 10775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:00:46.475998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33328 (local address 127.21.201.65:34503)
0812 01:00:46.476678 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0812 01:00:46.476687 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:00:46.477762 (+ 1075us) server_negotiation.cc:407] Connection header received
0812 01:00:46.478108 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:00:46.478118 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:00:46.479665 (+ 1547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:00:46.479910 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:00:46.481358 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.485027 (+ 3669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.490499 (+ 5472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.491642 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.491833 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:00:46.492137 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0812 01:00:46.492909 (+ 772us) server_negotiation.cc:299] Negotiation successful
0812 01:00:46.493093 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":335,"threads_started":1}
I20260812 01:00:46.494467 10777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:00:46.475913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:34503 (local address 127.0.0.1:33328)
0812 01:00:46.477595 (+ 1682us) negotiation.cc:107] Waiting for socket to connect
0812 01:00:46.477655 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 01:00:46.477846 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:00:46.480353 (+ 2507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:00:46.480364 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:00:46.480417 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:00:46.481070 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:00:46.481089 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.485273 (+ 4184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.485282 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:00:46.489936 (+ 4654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:00:46.489946 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.492281 (+ 2335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.492288 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:00:46.492524 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:00:46.492615 (+ 91us) client_negotiation.cc:769] Sending connection context
0812 01:00:46.492779 (+ 164us) client_negotiation.cc:240] Negotiation successful
0812 01:00:46.493403 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1363,"mutex_wait_us":46,"thread_start_us":1137,"threads_started":1}
I20260812 01:00:46.497941 10779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:00:46.477957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:37279 (local address 127.0.0.1:51104)
0812 01:00:46.488871 (+ 10914us) negotiation.cc:107] Waiting for socket to connect
0812 01:00:46.488887 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 01:00:46.489041 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:00:46.489615 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:00:46.489620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:00:46.489644 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:00:46.490387 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:00:46.490402 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.492649 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.492654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:00:46.495447 (+ 2793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:00:46.495462 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.496875 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.496882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:00:46.497051 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:00:46.497093 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 01:00:46.497203 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 01:00:46.497351 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10795,"mutex_wait_us":2,"thread_start_us":10606,"threads_started":1}
I20260812 01:00:46.498153 10778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:00:46.477206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60206 (local address 127.21.201.66:43573)
0812 01:00:46.488367 (+ 11161us) server_negotiation.cc:206] Beginning negotiation
0812 01:00:46.488375 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:00:46.488406 (+ 31us) server_negotiation.cc:407] Connection header received
0812 01:00:46.488500 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:00:46.488509 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:00:46.488806 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:00:46.489552 (+ 746us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:00:46.490232 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.492234 (+ 2002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.495447 (+ 3213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.496932 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.497160 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:00:46.497372 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0812 01:00:46.497556 (+ 184us) server_negotiation.cc:299] Negotiation successful
0812 01:00:46.497766 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10980,"thread_start_us":10860,"threads_started":1}
I20260812 01:00:46.499024 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:00:46.478397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51104 (local address 127.21.201.67:37279)
0812 01:00:46.488421 (+ 10024us) server_negotiation.cc:206] Beginning negotiation
0812 01:00:46.488432 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:00:46.488992 (+ 560us) server_negotiation.cc:407] Connection header received
0812 01:00:46.489209 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:00:46.489214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:00:46.489426 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:00:46.489619 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:00:46.490720 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.492508 (+ 1788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.495626 (+ 3118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.496661 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.496815 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:00:46.497054 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0812 01:00:46.498495 (+ 1441us) server_negotiation.cc:299] Negotiation successful
0812 01:00:46.498638 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9786,"thread_start_us":9123,"threads_started":1}
I20260812 01:00:46.498432 10774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:00:46.476529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:43573 (local address 127.0.0.1:60206)
0812 01:00:46.477639 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0812 01:00:46.477656 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:00:46.477836 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:00:46.489037 (+ 11201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:00:46.489042 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:00:46.489068 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:00:46.489784 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:00:46.489799 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.492381 (+ 2582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.492387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:00:46.495238 (+ 2851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:00:46.495255 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.497225 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.497231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:00:46.497381 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:00:46.497409 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 01:00:46.497510 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 01:00:46.497652 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":155,"thread_start_us":860,"threads_started":1}
I20260812 01:00:46.499184 10147 tablet_service.cc:1467] Processing CreateTablet for tablet c8f31f9b1c604f6385938ca8c09d59f3 (DEFAULT_TABLE table=test-table [id=c6ba2d4b5d594ecc9bf16fe7f24b2e71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:00:46.501538 10147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8f31f9b1c604f6385938ca8c09d59f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:00:46.503690 10413 tablet_service.cc:1467] Processing CreateTablet for tablet c8f31f9b1c604f6385938ca8c09d59f3 (DEFAULT_TABLE table=test-table [id=c6ba2d4b5d594ecc9bf16fe7f24b2e71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:00:46.503796 10641 tablet_service.cc:1467] Processing CreateTablet for tablet c8f31f9b1c604f6385938ca8c09d59f3 (DEFAULT_TABLE table=test-table [id=c6ba2d4b5d594ecc9bf16fe7f24b2e71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:00:46.505503 10413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8f31f9b1c604f6385938ca8c09d59f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:00:46.505525 10641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8f31f9b1c604f6385938ca8c09d59f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:00:46.513902 10782 tablet_bootstrap.cc:492] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac: Bootstrap starting.
I20260812 01:00:46.518149 10782 tablet_bootstrap.cc:654] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac: Neither blocks nor log segments found. Creating new log.
I20260812 01:00:46.519326 10782 log.cc:824] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac: Log is configured to *not* fsync() on all Append() calls
I20260812 01:00:46.522985 10782 tablet_bootstrap.cc:492] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac: No bootstrap required, opened a new log
I20260812 01:00:46.523270 10782 ts_tablet_manager.cc:1397] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260812 01:00:46.524637 10785 tablet_bootstrap.cc:492] T c8f31f9b1c604f6385938ca8c09d59f3 P 45bb05912e7c4882b6afc3ec126b7cbb: Bootstrap starting.
I20260812 01:00:46.525446 10784 tablet_bootstrap.cc:492] T c8f31f9b1c604f6385938ca8c09d59f3 P 34b33aeb9dd04a70a16cff2b843c1269: Bootstrap starting.
I20260812 01:00:46.528846 10784 tablet_bootstrap.cc:654] T c8f31f9b1c604f6385938ca8c09d59f3 P 34b33aeb9dd04a70a16cff2b843c1269: Neither blocks nor log segments found. Creating new log.
I20260812 01:00:46.530078 10784 log.cc:824] T c8f31f9b1c604f6385938ca8c09d59f3 P 34b33aeb9dd04a70a16cff2b843c1269: Log is configured to *not* fsync() on all Append() calls
I20260812 01:00:46.532611 10782 raft_consensus.cc:348] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bee060365b94dc5a752816a2e7ca6ac" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34503 } } peers { permanent_uuid: "45bb05912e7c4882b6afc3ec126b7cbb" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37279 } } peers { permanent_uuid: "34b33aeb9dd04a70a16cff2b843c1269" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 43573 } }
I20260812 01:00:46.533181 10784 tablet_bootstrap.cc:492] T c8f31f9b1c604f6385938ca8c09d59f3 P 34b33aeb9dd04a70a16cff2b843c1269: No bootstrap required, opened a new log
I20260812 01:00:46.533214 10782 raft_consensus.cc:374] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:00:46.533370 10782 raft_consensus.cc:729] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bee060365b94dc5a752816a2e7ca6ac, State: Initialized, Role: FOLLOWER
I20260812 01:00:46.533577 10784 ts_tablet_manager.cc:1397] T c8f31f9b1c604f6385938ca8c09d59f3 P 34b33aeb9dd04a70a16cff2b843c1269: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260812 01:00:46.534096 10785 tablet_bootstrap.cc:654] T c8f31f9b1c604f6385938ca8c09d59f3 P 45bb05912e7c4882b6afc3ec126b7cbb: Neither blocks nor log segments found. Creating new log.
I20260812 01:00:46.534368 10782 consensus_queue.cc:260] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bee060365b94dc5a752816a2e7ca6ac" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34503 } } peers { permanent_uuid: "45bb05912e7c4882b6afc3ec126b7cbb" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37279 } } peers { permanent_uuid: "34b33aeb9dd04a70a16cff2b843c1269" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 43573 } }
I20260812 01:00:46.535496 10785 log.cc:824] T c8f31f9b1c604f6385938ca8c09d59f3 P 45bb05912e7c4882b6afc3ec126b7cbb: Log is configured to *not* fsync() on all Append() calls
I20260812 01:00:46.536495 10782 ts_tablet_manager.cc:1428] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260812 01:00:46.538878 10785 tablet_bootstrap.cc:492] T c8f31f9b1c604f6385938ca8c09d59f3 P 45bb05912e7c4882b6afc3ec126b7cbb: No bootstrap required, opened a new log
I20260812 01:00:46.539155 10785 ts_tablet_manager.cc:1397] T c8f31f9b1c604f6385938ca8c09d59f3 P 45bb05912e7c4882b6afc3ec126b7cbb: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20260812 01:00:46.546159 10784 raft_consensus.cc:348] T c8f31f9b1c604f6385938ca8c09d59f3 P 34b33aeb9dd04a70a16cff2b843c1269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bee060365b94dc5a752816a2e7ca6ac" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34503 } } peers { permanent_uuid: "45bb05912e7c4882b6afc3ec126b7cbb" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37279 } } peers { permanent_uuid: "34b33aeb9dd04a70a16cff2b843c1269" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 43573 } }
I20260812 01:00:46.546626 10784 raft_consensus.cc:374] T c8f31f9b1c604f6385938ca8c09d59f3 P 34b33aeb9dd04a70a16cff2b843c1269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:00:46.546741 10784 raft_consensus.cc:729] T c8f31f9b1c604f6385938ca8c09d59f3 P 34b33aeb9dd04a70a16cff2b843c1269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34b33aeb9dd04a70a16cff2b843c1269, State: Initialized, Role: FOLLOWER
I20260812 01:00:46.547412 10784 consensus_queue.cc:260] T c8f31f9b1c604f6385938ca8c09d59f3 P 34b33aeb9dd04a70a16cff2b843c1269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bee060365b94dc5a752816a2e7ca6ac" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34503 } } peers { permanent_uuid: "45bb05912e7c4882b6afc3ec126b7cbb" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37279 } } peers { permanent_uuid: "34b33aeb9dd04a70a16cff2b843c1269" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 43573 } }
I20260812 01:00:46.548555 10515 heartbeater.cc:503] Master 127.21.201.126:34079 was elected leader, sending a full tablet report...
I20260812 01:00:46.549381 10784 ts_tablet_manager.cc:1428] T c8f31f9b1c604f6385938ca8c09d59f3 P 34b33aeb9dd04a70a16cff2b843c1269: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260812 01:00:46.550688 10785 raft_consensus.cc:348] T c8f31f9b1c604f6385938ca8c09d59f3 P 45bb05912e7c4882b6afc3ec126b7cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bee060365b94dc5a752816a2e7ca6ac" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34503 } } peers { permanent_uuid: "45bb05912e7c4882b6afc3ec126b7cbb" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37279 } } peers { permanent_uuid: "34b33aeb9dd04a70a16cff2b843c1269" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 43573 } }
I20260812 01:00:46.551237 10785 raft_consensus.cc:374] T c8f31f9b1c604f6385938ca8c09d59f3 P 45bb05912e7c4882b6afc3ec126b7cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:00:46.551362 10785 raft_consensus.cc:729] T c8f31f9b1c604f6385938ca8c09d59f3 P 45bb05912e7c4882b6afc3ec126b7cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45bb05912e7c4882b6afc3ec126b7cbb, State: Initialized, Role: FOLLOWER
I20260812 01:00:46.552042 10785 consensus_queue.cc:260] T c8f31f9b1c604f6385938ca8c09d59f3 P 45bb05912e7c4882b6afc3ec126b7cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bee060365b94dc5a752816a2e7ca6ac" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34503 } } peers { permanent_uuid: "45bb05912e7c4882b6afc3ec126b7cbb" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37279 } } peers { permanent_uuid: "34b33aeb9dd04a70a16cff2b843c1269" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 43573 } }
I20260812 01:00:46.553254 10752 heartbeater.cc:503] Master 127.21.201.126:34079 was elected leader, sending a full tablet report...
I20260812 01:00:46.555071 10785 ts_tablet_manager.cc:1428] T c8f31f9b1c604f6385938ca8c09d59f3 P 45bb05912e7c4882b6afc3ec126b7cbb: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260812 01:00:46.633754 10788 raft_consensus.cc:482] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:00:46.634162 10788 raft_consensus.cc:504] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bee060365b94dc5a752816a2e7ca6ac" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34503 } } peers { permanent_uuid: "45bb05912e7c4882b6afc3ec126b7cbb" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37279 } } peers { permanent_uuid: "34b33aeb9dd04a70a16cff2b843c1269" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 43573 } }
I20260812 01:00:46.635948 10788 leader_election.cc:288] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45bb05912e7c4882b6afc3ec126b7cbb (127.21.201.67:37279), 34b33aeb9dd04a70a16cff2b843c1269 (127.21.201.66:43573)
I20260812 01:00:46.643703 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:00:46.636355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:46031 (local address 127.21.201.67:37279)
0812 01:00:46.636565 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 01:00:46.636573 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:00:46.637549 (+ 976us) server_negotiation.cc:407] Connection header received
0812 01:00:46.637776 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:00:46.637782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:00:46.637874 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:00:46.637972 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:00:46.638560 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.639405 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.641590 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.642507 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.642714 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:00:46.642828 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 01:00:46.643223 (+ 395us) server_negotiation.cc:299] Negotiation successful
0812 01:00:46.643351 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 01:00:46.643784 10778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:00:46.636703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:38469 (local address 127.21.201.66:43573)
0812 01:00:46.636925 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0812 01:00:46.636949 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 01:00:46.638046 (+ 1097us) server_negotiation.cc:407] Connection header received
0812 01:00:46.638303 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:00:46.638308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:00:46.638367 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:00:46.638467 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:00:46.639111 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.639948 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.641833 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.642713 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.642881 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:00:46.642990 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 01:00:46.643338 (+ 348us) server_negotiation.cc:299] Negotiation successful
0812 01:00:46.643473 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 01:00:46.643961 10800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:00:46.636497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:43573 (local address 127.21.201.65:38469)
0812 01:00:46.637931 (+ 1434us) negotiation.cc:107] Waiting for socket to connect
0812 01:00:46.637946 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 01:00:46.638140 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:00:46.638483 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:00:46.638488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:00:46.638513 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:00:46.638947 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:00:46.638958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.640086 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.640092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:00:46.641679 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:00:46.641692 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.643033 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.643041 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:00:46.643183 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:00:46.643206 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 01:00:46.643310 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 01:00:46.643431 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1320,"mutex_wait_us":7,"thread_start_us":1124,"threads_started":1}
I20260812 01:00:46.644323 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:00:46.636290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:37279 (local address 127.21.201.65:46031)
0812 01:00:46.637422 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0812 01:00:46.637446 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 01:00:46.637629 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:00:46.637992 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:00:46.637996 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:00:46.638024 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:00:46.638417 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:00:46.638428 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.639537 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.639543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:00:46.641432 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:00:46.641449 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.642823 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.642830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:00:46.643029 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:00:46.643088 (+ 59us) client_negotiation.cc:769] Sending connection context
0812 01:00:46.643196 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 01:00:46.643336 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":992,"mutex_wait_us":22,"thread_start_us":777,"threads_started":1}
W20260812 01:00:46.645080 10754 tablet.cc:2369] T c8f31f9b1c604f6385938ca8c09d59f3 P 45bb05912e7c4882b6afc3ec126b7cbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:00:46.644974 10446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8f31f9b1c604f6385938ca8c09d59f3" candidate_uuid: "5bee060365b94dc5a752816a2e7ca6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34b33aeb9dd04a70a16cff2b843c1269" is_pre_election: true
I20260812 01:00:46.645685 10446 raft_consensus.cc:2454] T c8f31f9b1c604f6385938ca8c09d59f3 P 34b33aeb9dd04a70a16cff2b843c1269 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bee060365b94dc5a752816a2e7ca6ac in term 0.
I20260812 01:00:46.645612 10671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8f31f9b1c604f6385938ca8c09d59f3" candidate_uuid: "5bee060365b94dc5a752816a2e7ca6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45bb05912e7c4882b6afc3ec126b7cbb" is_pre_election: true
I20260812 01:00:46.646209 10671 raft_consensus.cc:2454] T c8f31f9b1c604f6385938ca8c09d59f3 P 45bb05912e7c4882b6afc3ec126b7cbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bee060365b94dc5a752816a2e7ca6ac in term 0.
I20260812 01:00:46.646515 10050 leader_election.cc:302] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34b33aeb9dd04a70a16cff2b843c1269, 5bee060365b94dc5a752816a2e7ca6ac; no voters:
I20260812 01:00:46.646989 10788 raft_consensus.cc:2781] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:00:46.647171 10788 raft_consensus.cc:482] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:00:46.647280 10788 raft_consensus.cc:3037] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:00:46.649756 10788 raft_consensus.cc:504] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bee060365b94dc5a752816a2e7ca6ac" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34503 } } peers { permanent_uuid: "45bb05912e7c4882b6afc3ec126b7cbb" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37279 } } peers { permanent_uuid: "34b33aeb9dd04a70a16cff2b843c1269" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 43573 } }
I20260812 01:00:46.650415 10788 leader_election.cc:288] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac [CANDIDATE]: Term 1 election: Requested vote from peers 45bb05912e7c4882b6afc3ec126b7cbb (127.21.201.67:37279), 34b33aeb9dd04a70a16cff2b843c1269 (127.21.201.66:43573)
I20260812 01:00:46.650852 10671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8f31f9b1c604f6385938ca8c09d59f3" candidate_uuid: "5bee060365b94dc5a752816a2e7ca6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45bb05912e7c4882b6afc3ec126b7cbb"
I20260812 01:00:46.650902 10446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8f31f9b1c604f6385938ca8c09d59f3" candidate_uuid: "5bee060365b94dc5a752816a2e7ca6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34b33aeb9dd04a70a16cff2b843c1269"
I20260812 01:00:46.651062 10671 raft_consensus.cc:3037] T c8f31f9b1c604f6385938ca8c09d59f3 P 45bb05912e7c4882b6afc3ec126b7cbb [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:00:46.651147 10446 raft_consensus.cc:3037] T c8f31f9b1c604f6385938ca8c09d59f3 P 34b33aeb9dd04a70a16cff2b843c1269 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:00:46.653476 10671 raft_consensus.cc:2454] T c8f31f9b1c604f6385938ca8c09d59f3 P 45bb05912e7c4882b6afc3ec126b7cbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bee060365b94dc5a752816a2e7ca6ac in term 1.
I20260812 01:00:46.653896 10046 leader_election.cc:302] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45bb05912e7c4882b6afc3ec126b7cbb, 5bee060365b94dc5a752816a2e7ca6ac; no voters:
I20260812 01:00:46.654271 10788 raft_consensus.cc:2781] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:00:46.654230 10446 raft_consensus.cc:2454] T c8f31f9b1c604f6385938ca8c09d59f3 P 34b33aeb9dd04a70a16cff2b843c1269 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bee060365b94dc5a752816a2e7ca6ac in term 1.
I20260812 01:00:46.655049 10788 raft_consensus.cc:686] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac [term 1 LEADER]: Becoming Leader. State: Replica: 5bee060365b94dc5a752816a2e7ca6ac, State: Running, Role: LEADER
I20260812 01:00:46.655623 10788 consensus_queue.cc:237] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bee060365b94dc5a752816a2e7ca6ac" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34503 } } peers { permanent_uuid: "45bb05912e7c4882b6afc3ec126b7cbb" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37279 } } peers { permanent_uuid: "34b33aeb9dd04a70a16cff2b843c1269" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 43573 } }
I20260812 01:00:46.660836 9854 catalog_manager.cc:5495] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bee060365b94dc5a752816a2e7ca6ac (127.21.201.65). New cstate: current_term: 1 leader_uuid: "5bee060365b94dc5a752816a2e7ca6ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bee060365b94dc5a752816a2e7ca6ac" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 34503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45bb05912e7c4882b6afc3ec126b7cbb" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34b33aeb9dd04a70a16cff2b843c1269" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 43573 } health_report { overall_health: UNKNOWN } } }
I20260812 01:00:46.681210 10775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:00:46.675657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33338 (local address 127.21.201.65:34503)
0812 01:00:46.675867 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 01:00:46.675872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:00:46.676027 (+ 155us) server_negotiation.cc:407] Connection header received
0812 01:00:46.676246 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:00:46.676254 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:00:46.676343 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:00:46.676481 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:00:46.676872 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.677696 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.679086 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.679570 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.679664 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:00:46.680239 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:00:46.680744 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:00:46.680868 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 01:00:46.680933 (+ 65us) server_negotiation.cc:299] Negotiation successful
0812 01:00:46.681001 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260812 01:00:46.689245 10519 tablet.cc:2369] T c8f31f9b1c604f6385938ca8c09d59f3 P 34b33aeb9dd04a70a16cff2b843c1269: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:00:46.690994 10446 raft_consensus.cc:1261] T c8f31f9b1c604f6385938ca8c09d59f3 P 34b33aeb9dd04a70a16cff2b843c1269 [term 1 FOLLOWER]: Refusing update from remote peer 5bee060365b94dc5a752816a2e7ca6ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:00:46.691275 10671 raft_consensus.cc:1261] T c8f31f9b1c604f6385938ca8c09d59f3 P 45bb05912e7c4882b6afc3ec126b7cbb [term 1 FOLLOWER]: Refusing update from remote peer 5bee060365b94dc5a752816a2e7ca6ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:00:46.692102 10788 consensus_queue.cc:1035] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "34b33aeb9dd04a70a16cff2b843c1269" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 43573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:00:46.692533 10801 consensus_queue.cc:1035] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "45bb05912e7c4882b6afc3ec126b7cbb" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:00:46.706434 10807 mvcc.cc:205] Tried to move back new op lower bound from 7317489445635403776 to 7317489445505490944. Current Snapshot: MvccSnapshot[applied={T|T < 7317489445635403776}]
I20260812 01:00:46.707496 10812 mvcc.cc:205] Tried to move back new op lower bound from 7317489445635403776 to 7317489445505490944. Current Snapshot: MvccSnapshot[applied={T|T < 7317489445635403776}]
I20260812 01:00:46.728367 9854 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:00:46.728741 9854 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:00:46.733908 9854 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7cc48f8c8c92431aaa088f6eb0022bb4: Sending DeleteTablet for 3 replicas of tablet c8f31f9b1c604f6385938ca8c09d59f3
I20260812 01:00:46.735525 10413 tablet_service.cc:1514] Processing DeleteTablet for tablet c8f31f9b1c604f6385938ca8c09d59f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:00:46 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:60206
I20260812 01:00:46.736423 10147 tablet_service.cc:1514] Processing DeleteTablet for tablet c8f31f9b1c604f6385938ca8c09d59f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:00:46 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:33328
I20260812 01:00:46.737656 10641 tablet_service.cc:1514] Processing DeleteTablet for tablet c8f31f9b1c604f6385938ca8c09d59f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:00:46 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:51104
I20260812 01:00:46.741264 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 10006
I20260812 01:00:46.744302 10834 tablet_replica.cc:330] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac: stopping tablet replica
I20260812 01:00:46.744819 10834 raft_consensus.cc:2229] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:00:46.745739 10834 raft_consensus.cc:2258] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:00:46.748550 10834 ts_tablet_manager.cc:1905] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:00:46.754436 10828 tablet_replica.cc:330] T c8f31f9b1c604f6385938ca8c09d59f3 P 34b33aeb9dd04a70a16cff2b843c1269: stopping tablet replica
I20260812 01:00:46.755043 10828 raft_consensus.cc:2229] T c8f31f9b1c604f6385938ca8c09d59f3 P 34b33aeb9dd04a70a16cff2b843c1269 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:00:46.755718 10828 raft_consensus.cc:2258] T c8f31f9b1c604f6385938ca8c09d59f3 P 34b33aeb9dd04a70a16cff2b843c1269 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:00:46.756688 10835 tablet_replica.cc:330] T c8f31f9b1c604f6385938ca8c09d59f3 P 45bb05912e7c4882b6afc3ec126b7cbb: stopping tablet replica
I20260812 01:00:46.757077 10835 raft_consensus.cc:2229] T c8f31f9b1c604f6385938ca8c09d59f3 P 45bb05912e7c4882b6afc3ec126b7cbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:00:46.757738 10835 raft_consensus.cc:2258] T c8f31f9b1c604f6385938ca8c09d59f3 P 45bb05912e7c4882b6afc3ec126b7cbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:00:46.759325 10834 ts_tablet_manager.cc:1918] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:00:46.759588 10834 log.cc:1196] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-0/wal/wals/c8f31f9b1c604f6385938ca8c09d59f3
I20260812 01:00:46.760375 10828 ts_tablet_manager.cc:1905] T c8f31f9b1c604f6385938ca8c09d59f3 P 34b33aeb9dd04a70a16cff2b843c1269: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:00:46.761492 10835 ts_tablet_manager.cc:1905] T c8f31f9b1c604f6385938ca8c09d59f3 P 45bb05912e7c4882b6afc3ec126b7cbb: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 12 01:00:46 dist-test-slave-7f01 krb5kdc[8969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.65@KRBTEST.COM
I20260812 01:00:46.768597 10828 ts_tablet_manager.cc:1918] T c8f31f9b1c604f6385938ca8c09d59f3 P 34b33aeb9dd04a70a16cff2b843c1269: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:00:46.768762 10828 log.cc:1196] T c8f31f9b1c604f6385938ca8c09d59f3 P 34b33aeb9dd04a70a16cff2b843c1269: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/c8f31f9b1c604f6385938ca8c09d59f3
I20260812 01:00:46.769356 10828 ts_tablet_manager.cc:1939] T c8f31f9b1c604f6385938ca8c09d59f3 P 34b33aeb9dd04a70a16cff2b843c1269: Deleting consensus metadata
I20260812 01:00:46.769536 10835 ts_tablet_manager.cc:1918] T c8f31f9b1c604f6385938ca8c09d59f3 P 45bb05912e7c4882b6afc3ec126b7cbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:00:46.769760 10835 log.cc:1196] T c8f31f9b1c604f6385938ca8c09d59f3 P 45bb05912e7c4882b6afc3ec126b7cbb: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/c8f31f9b1c604f6385938ca8c09d59f3
I20260812 01:00:46.770294 10835 ts_tablet_manager.cc:1939] T c8f31f9b1c604f6385938ca8c09d59f3 P 45bb05912e7c4882b6afc3ec126b7cbb: Deleting consensus metadata
I20260812 01:00:46.771817 9836 catalog_manager.cc:4841] TS 34b33aeb9dd04a70a16cff2b843c1269 (127.21.201.66:43573): tablet c8f31f9b1c604f6385938ca8c09d59f3 (table test-table [id=c6ba2d4b5d594ecc9bf16fe7f24b2e71]) successfully deleted
I20260812 01:00:46.772154 9830 catalog_manager.cc:4841] TS 45bb05912e7c4882b6afc3ec126b7cbb (127.21.201.67:37279): tablet c8f31f9b1c604f6385938ca8c09d59f3 (table test-table [id=c6ba2d4b5d594ecc9bf16fe7f24b2e71]) successfully deleted
I20260812 01:00:46.772408 10834 ts_tablet_manager.cc:1939] T c8f31f9b1c604f6385938ca8c09d59f3 P 5bee060365b94dc5a752816a2e7ca6ac: Deleting consensus metadata
I20260812 01:00:46.774286 9830 catalog_manager.cc:4841] TS 5bee060365b94dc5a752816a2e7ca6ac (127.21.201.65:34503): tablet c8f31f9b1c604f6385938ca8c09d59f3 (table test-table [id=c6ba2d4b5d594ecc9bf16fe7f24b2e71]) successfully deleted
I20260812 01:00:46.776315 10775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:00:46.742798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33346 (local address 127.21.201.65:34503)
0812 01:00:46.743013 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 01:00:46.743020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:00:46.753422 (+ 10402us) server_negotiation.cc:407] Connection header received
0812 01:00:46.753661 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:00:46.753666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:00:46.753754 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:00:46.753869 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:00:46.756289 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.757510 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.759533 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.760371 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.760484 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:00:46.766572 (+ 6088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:00:46.766592 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:00:46.766611 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:00:46.766670 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:00:46.771481 (+ 4811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:00:46.773460 (+ 1979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:00:46.773479 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:00:46.773483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:00:46.773799 (+ 316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:00:46.774309 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:00:46.774316 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:00:46.774319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:00:46.774852 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:00:46.775298 (+ 446us) server_negotiation.cc:1035] Waiting for connection context
0812 01:00:46.775544 (+ 246us) server_negotiation.cc:299] Negotiation successful
0812 01:00:46.775869 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 01:00:46.777117 10267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:00:46.877467 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 10006
I20260812 01:00:46.908607 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 10285
Aug 12 01:00:46 dist-test-slave-7f01 krb5kdc[8969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.66@KRBTEST.COM
I20260812 01:00:46.925081 10778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:00:46.909544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60214 (local address 127.21.201.66:43573)
0812 01:00:46.909753 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 01:00:46.909761 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:00:46.909788 (+ 27us) server_negotiation.cc:407] Connection header received
0812 01:00:46.909872 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:00:46.909877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:00:46.909938 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:00:46.910057 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:00:46.911454 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.912330 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.913716 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:46.914118 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:46.914264 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:00:46.918878 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:00:46.918893 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:00:46.918910 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:00:46.918973 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:00:46.921978 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:00:46.922691 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:00:46.922699 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:00:46.922701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:00:46.922841 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:00:46.923285 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:00:46.923289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:00:46.923291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:00:46.923950 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:00:46.924203 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0812 01:00:46.924633 (+ 430us) server_negotiation.cc:299] Negotiation successful
0812 01:00:46.924798 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 01:00:46.925808 10508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:00:47.017297 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 10285
I20260812 01:00:47.045017 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 10524
Aug 12 01:00:47 dist-test-slave-7f01 krb5kdc[8969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.67@KRBTEST.COM
I20260812 01:00:47.065389 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:00:47.045786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51108 (local address 127.21.201.67:37279)
0812 01:00:47.045966 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 01:00:47.045970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:00:47.045989 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:00:47.046110 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:00:47.046114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:00:47.046177 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:00:47.046265 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:00:47.047650 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:47.048541 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:47.049795 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:00:47.050257 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:00:47.050383 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:00:47.055796 (+ 5413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:00:47.055822 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:00:47.055849 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:00:47.055911 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:00:47.059933 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:00:47.061019 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:00:47.061034 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:00:47.061039 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:00:47.061226 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:00:47.061712 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:00:47.061718 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:00:47.061721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:00:47.062211 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:00:47.062533 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0812 01:00:47.064946 (+ 2413us) server_negotiation.cc:299] Negotiation successful
0812 01:00:47.065120 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 01:00:47.066273 10746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:00:47.165989 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 10524
I20260812 01:00:47.192380 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 9795
I20260812 01:00:47.193671 9902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:00:47.311533 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 9795
I20260812 01:00:47.343442 22309 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 01:00:47.689070 22309 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 01:00:47.689440 22309 mini_ranger.cc:66] Stopping Ranger...
I20260812 01:00:48.035128 22309 mini_ranger.cc:68] Stopped Ranger
2026-08-12 01:00:48.035 UTC [9034] LOG: received smart shutdown request
2026-08-12 01:00:48.039 UTC [9034] LOG: background worker "logical replication launcher" (PID 9045) exited with exit code 1
2026-08-12 01:00:48.039 UTC [9038] LOG: shutting down
2026-08-12 01:00:48.202 UTC [9034] LOG: database system is shut down
2026-08-12T01:00:48Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52945 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:00:48 dist-test-slave-7f01 krb5kdc[10855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:00:48 dist-test-slave-7f01 krb5kdc[10855](info): set up 2 sockets
Aug 12 01:00:48 dist-test-slave-7f01 krb5kdc[10855](info): commencing operation
krb5kdc: starting...
W20260812 01:00:49.701583 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.317s user 0.002s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:00:49 dist-test-slave-7f01 krb5kdc[10855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:00:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:00:49Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.201.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.201.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.201.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/rangeradmin_127.21.201.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.201.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/rangeradmin_127.21.201.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.201.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.201.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.201.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/rangerlookup_127.21.201.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.201.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/rangerlookup_127.21.201.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.201.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.201.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.201.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/HTTP_127.21.201.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.201.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/HTTP_127.21.201.84@KRBTEST.COM.keytab.
I20260812 01:00:50.110167 22309 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskJMOGj2/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/postgres -l logfile start
2026-08-12 01:00:52.655 UTC [10905] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 01:00:52.655 UTC [10905] LOG: listening on IPv4 address "127.21.201.84", port 51517
2026-08-12 01:00:52.659 UTC [10905] LOG: listening on Unix socket "/tmp/.s.PGSQL.51517"
2026-08-12 01:00:52.683 UTC [10908] LOG: database system was shut down at 2026-08-12 01:00:51 UTC
2026-08-12 01:00:52.687 UTC [10905] LOG: database system is ready to accept connections
I20260812 01:00:53.919688 22309 mini_postgres.cc:96] Postgres bound to 51517
2026-08-12 01:00:53.943 UTC [10921] FATAL: database "slave" does not exist
127.21.201.84:51517 - accepting connections
I20260812 01:00:53.943892 22309 mini_ranger.cc:161] Starting Ranger...
I20260812 01:00:53.975948 22309 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 01:00:54.468132 22309 mini_ranger.cc:87] Created ranger Postgres database
I20260812 01:00:54.468268 22309 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ranger-admin
2026-08-12 01:00:54,787 [I] DB FLAVOR :POSTGRES
2026-08-12 01:00:54,788 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 01:00:54,788 [I] Checking connection..
2026-08-12 01:00:54,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:51517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 01:00:55,063 [I] Checking connection passed.
2026-08-12 01:00:55,064 [I] --------- Verifying version history table ---------
2026-08-12 01:00:55,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:51517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 01:00:55,314 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 01:00:55,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:51517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 01:00:55,563 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 01:00:55,564 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 01:00:55,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:51517/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 01:00:55.831 UTC [11036] WARNING: there is no transaction in progress
2026-08-12 01:00:55,840 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 01:00:55,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:51517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 01:00:56,122 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 01:00:56,123 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 01:00:56,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:51517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 01:00:56,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:51517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 01:00:56,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:51517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7f01.c.gcp-upstream.internal','N') ;"
2026-08-12 01:00:56,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:51517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 01:00:57,218 [I] Table x_portal_user does not exist in database ranger
2026-08-12 01:00:57,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:51517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 01:00:57,506 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 01:00:57,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:51517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 01:00:57,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:51517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 01:00:58,043 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 01:00:58,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:51517/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 01:00:59.073 UTC [11250] WARNING: there is no transaction in progress
2026-08-12 01:00:59.088 UTC [11250] WARNING: there is no transaction in progress
2026-08-12 01:00:59.103 UTC [11250] WARNING: there is no transaction in progress
2026-08-12 01:00:59.118 UTC [11250] WARNING: there is no transaction in progress
2026-08-12 01:00:59.134 UTC [11250] WARNING: there is no transaction in progress
2026-08-12 01:00:59.155 UTC [11250] WARNING: there is no transaction in progress
2026-08-12 01:00:59.207 UTC [11250] WARNING: there is no transaction in progress
2026-08-12 01:00:59.214 UTC [11250] WARNING: there is no transaction in progress
2026-08-12 01:00:59.228 UTC [11250] WARNING: there is no transaction in progress
2026-08-12 01:00:59.237 UTC [11250] WARNING: there is no transaction in progress
2026-08-12 01:00:59.247 UTC [11250] WARNING: there is no transaction in progress
2026-08-12 01:00:59.261 UTC [11250] WARNING: there is no transaction in progress
2026-08-12 01:00:59.271 UTC [11250] WARNING: there is no transaction in progress
2026-08-12 01:00:59.277 UTC [11250] WARNING: there is no transaction in progress
2026-08-12 01:00:59.284 UTC [11250] WARNING: there is no transaction in progress
2026-08-12 01:00:59.290 UTC [11250] WARNING: there is no transaction in progress
2026-08-12 01:00:59,937 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 01:00:59,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:51517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 01:01:00,185 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 01:01:00,186 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:51517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 01:01:00,459 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 01:01:00,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:51517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 01:01:00,738 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 01:01:00,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:51517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 01:01:01,009 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:51517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 01:01:01,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:51517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7f01.c.gcp-upstream.internal';"
2026-08-12 01:01:01,531 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 01:01:01,532 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:51517/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 01:01:01,805 [I] DB_PATCHES have already been applied
I20260812 01:01:01.811827 22309 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ranger-admin:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskJMOGj2/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 01:01:01.811955 22309 mini_ranger.cc:193] Using host: 127.21.201.84
I20260812 01:01:01.832454 22309 mini_ranger.cc:227] Ranger admin URL: http://127.21.201.84:36945
Aug 12, 2026 1:01:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 1:01:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 1:01:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 1:01:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 1:01:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 1:01:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 1:01:02 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 01:01:03.307439 22309 mini_ranger.cc:160] Time spent starting Ranger: real 9.364s user 0.007s sys 0.052s
Aug 12, 2026 1:01:23 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 1:01:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 12, 2026 1:01:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 1:01:23 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 1:01:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 1:01:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 1:01:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 1:01:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 1:01:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 1:01:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 1:01:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 1:01:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 1:01:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 1:01:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 1:01:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 1:01:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 1:01:23 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 1:01:24 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 01:01:25.617327 22309 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.201.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.201.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.201.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/rangerkms_127.21.201.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.201.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/rangerkms_127.21.201.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.201.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.201.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.201.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/HTTP_127.21.201.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.201.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/HTTP_127.21.201.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 01:01:25 dist-test-slave-7f01 krb5kdc[10855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496485, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 01:01:25.852734 22309 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 01:01:25.887949 22309 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 01:01:26.735598 22309 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 01:01:26.735733 22309 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ranger-kms
I20260812 01:01:26.735833 22309 mini_ranger_kms.cc:227] Using postgres at 127.21.201.84:51517
2026-08-12 01:01:27,132 [I] DB FLAVOR :POSTGRES
2026-08-12 01:01:27,132 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 01:01:27,133 [I] Checking connection
2026-08-12 01:01:27,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:51517/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 01:01:27,400 [I] connection success
2026-08-12 01:01:27,400 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 01:01:27,400 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 01:01:27,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:51517/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 01:01:27,670 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 01:01:27,671 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 01:01:27,671 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 01:01:27,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.201.84:51517/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 01:01:27,938 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 01:01:28.447352 22309 mini_ranger_kms.cc:310] Created kms service
I20260812 01:01:28.674424 22309 mini_ranger_kms.cc:326] Created kudu user
I20260812 01:01:28.726758 22309 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 01:01:28 dist-test-slave-7f01 krb5kdc[10855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496485, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.201.84@KRBTEST.COM
I20260812 01:01:29.162390 22309 mini_ranger_kms.cc:384] Added ranger policy
I20260812 01:01:29.162745 22309 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ranger-kms:/tmp/dist-test-taskJMOGj2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskJMOGj2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 01:01:29.162811 22309 mini_ranger_kms.cc:240] Using host: 127.21.201.84
I20260812 01:01:29.184420 22309 mini_ranger_kms.cc:276] Ranger KMS PID: 11587
I20260812 01:01:29.184558 22309 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.201.84:57545
Aug 12, 2026 1:01:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 1:01:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 1:01:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 1:01:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 1:01:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 1:01:30 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 01:01:31 dist-test-slave-7f01 krb5kdc[10855](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786496491, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.201.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:01:32.248678 22309 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.396s user 0.077s sys 0.353s
I20260812 01:01:32.249001 22309 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 01:01:32.249214 22309 mini_ranger_kms.cc:401] 127.21.201.84:57545/kms/v1/keys
Aug 12 01:01:32 dist-test-slave-7f01 krb5kdc[10855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496491, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.201.84@KRBTEST.COM for HTTP/127.21.201.84@KRBTEST.COM
Aug 12 01:01:32 dist-test-slave-7f01 krb5kdc[10855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496491, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.201.84@KRBTEST.COM for HTTP/127.21.201.84@KRBTEST.COM
Aug 12 01:01:35 dist-test-slave-7f01 krb5kdc[10855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 01:01:35.200325 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:38675
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:44549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:38675
--ranger_config_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.201.84:57545/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:01:35.556147 11683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:01:35.556456 11683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:01:35.556555 11683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:01:35.564016 11683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:01:35.564124 11683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:01:35.564170 11683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:01:35.564229 11683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:01:35.564280 11683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:01:35.574318 11683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:44549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.201.84:57545/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:38675
--ranger_config_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:38675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:01:35.575603 11683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:01:35.576972 11683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:01:35.589285 11690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:35.589313 11691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:35.589404 11694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:01:35.590011 11683 server_base.cc:1034] running on GCE node
I20260812 01:01:35.590865 11683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:01:35.592442 11683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:01:35.593676 11683 hybrid_clock.cc:648] HybridClock initialized: now 1786496495593648 us; error 48 us; skew 500 ppm
I20260812 01:01:35.594029 11683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:01:35 dist-test-slave-7f01 krb5kdc[10855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496495, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:01:35.601379 11683 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 01:01:35.602893 11683 webserver.cc:460] Webserver started at http://127.21.201.126:46843/ using document root <none> and password file <none>
I20260812 01:01:35.603510 11683 fs_manager.cc:359] Metadata directory not provided
I20260812 01:01:35.603610 11683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:01:35.603950 11683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 01:01:35 dist-test-slave-7f01 krb5kdc[10855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496495, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for HTTP/127.21.201.84@KRBTEST.COM
I20260812 01:01:35.665421 11683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "7e0928e827fe43ae9ae7af7ca7fbb828"
format_stamp: "Formatted at 2026-08-12 01:01:35 on dist-test-slave-7f01"
server_key: "f47fc8f04eea86bceb3e35502104f2cf"
server_key_iv: "c74fd1ee42ecdecba0236932e666aa59"
server_key_version: "kuduclusterkey@0"
I20260812 01:01:35.666579 11683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "7e0928e827fe43ae9ae7af7ca7fbb828"
format_stamp: "Formatted at 2026-08-12 01:01:35 on dist-test-slave-7f01"
server_key: "f47fc8f04eea86bceb3e35502104f2cf"
server_key_iv: "c74fd1ee42ecdecba0236932e666aa59"
server_key_version: "kuduclusterkey@0"
I20260812 01:01:35.673801 11683 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260812 01:01:35.694630 11708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:35.696414 11683 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 01:01:35.696616 11683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "7e0928e827fe43ae9ae7af7ca7fbb828"
format_stamp: "Formatted at 2026-08-12 01:01:35 on dist-test-slave-7f01"
server_key: "f47fc8f04eea86bceb3e35502104f2cf"
server_key_iv: "c74fd1ee42ecdecba0236932e666aa59"
server_key_version: "kuduclusterkey@0"
I20260812 01:01:35.696885 11683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:01:35.724129 11683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:01:35.728991 11683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:01:35.729287 11683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:01:35.749809 11683 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:38675
I20260812 01:01:35.749840 11816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:38675 every 8 connection(s)
I20260812 01:01:35.751235 11683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:01:35.755179 11817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:01:35.760598 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 11683
I20260812 01:01:35.760828 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:01:35.761571 22309 external_mini_cluster.cc:1428] Setting key de55e2da64c0ac96c1141f7a0b2ed8e5
I20260812 01:01:35.767001 11817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828: Bootstrap starting.
I20260812 01:01:35.770820 11817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828: Neither blocks nor log segments found. Creating new log.
I20260812 01:01:35.772034 11817 log.cc:824] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828: Log is configured to *not* fsync() on all Append() calls
I20260812 01:01:35.775079 11817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828: No bootstrap required, opened a new log
Aug 12 01:01:35 dist-test-slave-7f01 krb5kdc[10855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:01:35.785625 11817 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0928e827fe43ae9ae7af7ca7fbb828" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38675 } }
I20260812 01:01:35.786065 11817 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:01:35.786171 11817 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e0928e827fe43ae9ae7af7ca7fbb828, State: Initialized, Role: FOLLOWER
I20260812 01:01:35.786867 11817 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0928e827fe43ae9ae7af7ca7fbb828" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38675 } }
I20260812 01:01:35.787082 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:35.763702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38606 (local address 127.21.201.126:38675)
0812 01:01:35.764653 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:35.764670 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:35.764720 (+ 50us) server_negotiation.cc:407] Connection header received
0812 01:01:35.767111 (+ 2391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:35.767138 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:35.767624 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:35.768056 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:35.769291 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:35.771324 (+ 2033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:35.772925 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:35.773770 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:35.773959 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:35.778109 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:35.778126 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:35.778151 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:35.778195 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:35.782327 (+ 4132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:35.783271 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:35.783279 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:35.783288 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:35.783522 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:35.783963 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:35.783971 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:35.783974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:35.784709 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:35.785016 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:35.785346 (+ 330us) server_negotiation.cc:299] Negotiation successful
0812 01:01:35.785743 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":377,"threads_started":1}
I20260812 01:01:35.787173 11817 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:01:35.787307 11817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:01:35.787516 11817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:01:35.790431 11817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0928e827fe43ae9ae7af7ca7fbb828" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38675 } }
I20260812 01:01:35.790863 11817 leader_election.cc:302] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e0928e827fe43ae9ae7af7ca7fbb828; no voters:
I20260812 01:01:35.791394 11817 leader_election.cc:288] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:01:35.791565 11827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:01:35.792870 11827 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [term 1 LEADER]: Becoming Leader. State: Replica: 7e0928e827fe43ae9ae7af7ca7fbb828, State: Running, Role: LEADER
I20260812 01:01:35.793362 11827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0928e827fe43ae9ae7af7ca7fbb828" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38675 } }
I20260812 01:01:35.793817 11817 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:01:35.796257 11832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e0928e827fe43ae9ae7af7ca7fbb828. Latest consensus state: current_term: 1 leader_uuid: "7e0928e827fe43ae9ae7af7ca7fbb828" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0928e827fe43ae9ae7af7ca7fbb828" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38675 } } }
I20260812 01:01:35.797178 11832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [sys.catalog]: This master's current role is: LEADER
I20260812 01:01:35.796330 11833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e0928e827fe43ae9ae7af7ca7fbb828" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0928e827fe43ae9ae7af7ca7fbb828" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38675 } } }
I20260812 01:01:35.797703 11833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [sys.catalog]: This master's current role is: LEADER
I20260812 01:01:35.799454 11840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:01:35.805403 11840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:01:35.806238 11817 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 01:01:35.823366 11840 catalog_manager.cc:1347] Generated new cluster ID: 00368148a6d44611b570a9a169145fc2
I20260812 01:01:35.823532 11840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:01:35.844920 11840 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:01:35.847519 11840 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:01:35.860634 11840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828: Generated new TSK 0
I20260812 01:01:35.861747 11840 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.201.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
I20260812 01:01:36.624228 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.201.84:57545/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:0
--local_ip_for_outbound_sockets=127.21.201.65
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38675
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:44549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 01:01:36 dist-test-slave-7f01 krb5kdc[10855](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786496496, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 01:01:37.072628 11904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:01:37.073036 11904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:01:37.073267 11904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:01:37.084983 11904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:01:37.085136 11904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:01:37.085378 11904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 01:01:37.100988 11904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:44549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.201.84:57545/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:01:37.102903 11904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:01:37.104665 11904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:01:37.129582 11925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:37.129545 11926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:37.130048 11929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:01:37.132143 11904 server_base.cc:1034] running on GCE node
I20260812 01:01:37.133021 11904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:01:37.134385 11904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:01:37.135717 11904 hybrid_clock.cc:648] HybridClock initialized: now 1786496497135615 us; error 87 us; skew 500 ppm
I20260812 01:01:37.136188 11904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:01:37 dist-test-slave-7f01 krb5kdc[10855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496497, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:01:37.140941 11904 init.cc:377] Logged in from keytab as kudu/127.21.201.65@KRBTEST.COM (short username kudu)
I20260812 01:01:37.143097 11904 webserver.cc:460] Webserver started at http://127.21.201.65:38401/ using document root <none> and password file <none>
I20260812 01:01:37.144150 11904 fs_manager.cc:359] Metadata directory not provided
I20260812 01:01:37.144305 11904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:01:37.144675 11904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 01:01:37 dist-test-slave-7f01 krb5kdc[10855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496497, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for HTTP/127.21.201.84@KRBTEST.COM
I20260812 01:01:37.163339 11904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/data/instance:
uuid: "471561b216ca438bb4464871c8a516b3"
format_stamp: "Formatted at 2026-08-12 01:01:37 on dist-test-slave-7f01"
server_key: "58fcdc07cf837fbedd91fc9952dd3a9d"
server_key_iv: "c93b84d9e4755f9f30cf558a6136d022"
server_key_version: "kuduclusterkey@0"
I20260812 01:01:37.165053 11904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance:
uuid: "471561b216ca438bb4464871c8a516b3"
format_stamp: "Formatted at 2026-08-12 01:01:37 on dist-test-slave-7f01"
server_key: "58fcdc07cf837fbedd91fc9952dd3a9d"
server_key_iv: "c93b84d9e4755f9f30cf558a6136d022"
server_key_version: "kuduclusterkey@0"
I20260812 01:01:37.177645 11904 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.003s sys 0.008s
I20260812 01:01:37.216421 11939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:37.218549 11904 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.009s
I20260812 01:01:37.218792 11904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "471561b216ca438bb4464871c8a516b3"
format_stamp: "Formatted at 2026-08-12 01:01:37 on dist-test-slave-7f01"
server_key: "58fcdc07cf837fbedd91fc9952dd3a9d"
server_key_iv: "c93b84d9e4755f9f30cf558a6136d022"
server_key_version: "kuduclusterkey@0"
I20260812 01:01:37.219108 11904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:01:37.253925 11904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:01:37.259423 11904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:01:37.259819 11904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:01:37.261099 11904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:01:37.263309 11904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:01:37.263417 11904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:37.263541 11904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:01:37.263594 11904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:37.331281 11904 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:43425
I20260812 01:01:37.331286 12142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:43425 every 8 connection(s)
I20260812 01:01:37.333091 11904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 01:01:37.335418 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 11904
I20260812 01:01:37.335691 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance
I20260812 01:01:37.336601 22309 external_mini_cluster.cc:1428] Setting key 72d6f62de5a95594f7bbd6b378f710b7
Aug 12 01:01:37 dist-test-slave-7f01 krb5kdc[10855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496497, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
Aug 12 01:01:37 dist-test-slave-7f01 krb5kdc[10855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496496, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for HTTP/127.21.201.84@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.201.66@KRBTEST.COM; defaulting to no policy
I20260812 01:01:37.398077 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:37.342216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:40473 (local address 127.21.201.126:38675)
0812 01:01:37.353428 (+ 11212us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:37.353435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:37.365231 (+ 11796us) server_negotiation.cc:407] Connection header received
0812 01:01:37.377779 (+ 12548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:37.377787 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:37.377911 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:37.378017 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:37.380558 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:37.381760 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:37.384206 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:37.384834 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:37.384938 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:37.389632 (+ 4694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:37.389640 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:37.389644 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:37.389687 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:37.393692 (+ 4005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:37.395007 (+ 1315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:37.395014 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:37.395019 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:37.395180 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:37.395793 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:37.395799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:37.395802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:37.396274 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:37.396416 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:37.397460 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0812 01:01:37.397700 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11047,"thread_start_us":10913,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.66@KRBTEST.COM" created.
I20260812 01:01:37.400678 12149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:37.342824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38675 (local address 127.21.201.65:40473)
0812 01:01:37.365031 (+ 22207us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:37.365084 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:37.377352 (+ 12268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:37.378696 (+ 1344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:37.378711 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:37.379400 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:01:37.380305 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:37.380331 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:37.381974 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:37.381979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:37.384033 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:37.384048 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:37.385335 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:37.385345 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:37.385655 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:37.386406 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:01:37.386437 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:01:37.389349 (+ 2912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:01:37.393877 (+ 4528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:37.393887 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:37.393910 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:37.394682 (+ 772us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:37.395326 (+ 644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:37.395332 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:37.395336 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:37.395631 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:37.396548 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:01:37.396556 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:01:37.397026 (+ 470us) client_negotiation.cc:769] Sending connection context
0812 01:01:37.398643 (+ 1617us) client_negotiation.cc:240] Negotiation successful
0812 01:01:37.399189 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21894,"thread_start_us":21655,"threads_started":1}
I20260812 01:01:37.403079 12145 heartbeater.cc:347] Connected to a master server at 127.21.201.126:38675
I20260812 01:01:37.403411 12145 heartbeater.cc:464] Registering TS with master...
I20260812 01:01:37.404296 12145 heartbeater.cc:511] Master 127.21.201.126:38675 requested a full tablet report, sending...
I20260812 01:01:37.406874 11745 ts_manager.cc:194] Registered new tserver with Master: 471561b216ca438bb4464871c8a516b3 (127.21.201.65:43425)
I20260812 01:01:37.408809 11745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.65@KRBTEST.COM'} at 127.21.201.65:40473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
W20260812 01:01:37.443011 22309 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.21.201.66: real 0.103s user 0.003s sys 0.021s
I20260812 01:01:37.444186 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.201.84:57545/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:0
--local_ip_for_outbound_sockets=127.21.201.66
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38675
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:44549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 01:01:37 dist-test-slave-7f01 krb5kdc[10855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496496, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for HTTP/127.21.201.84@KRBTEST.COM
W20260812 01:01:37.969142 12157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:01:37.969502 12157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:01:37.969691 12157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:01:37.980849 12157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:01:37.981045 12157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:01:37.981254 12157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 01:01:37.994611 12157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:44549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.201.84:57545/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:01:37.996330 12157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:01:37.997759 12157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:01:38.018249 12168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:38.018572 12169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:01:38.034011 12157 server_base.cc:1034] running on GCE node
W20260812 01:01:38.034919 12173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:01:38.036469 12157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:01:38.039752 12157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:01:38.043182 12157 hybrid_clock.cc:648] HybridClock initialized: now 1786496498043125 us; error 41 us; skew 500 ppm
I20260812 01:01:38.043609 12157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:01:38 dist-test-slave-7f01 krb5kdc[10855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496498, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:01:38.048283 12157 init.cc:377] Logged in from keytab as kudu/127.21.201.66@KRBTEST.COM (short username kudu)
I20260812 01:01:38.050264 12157 webserver.cc:460] Webserver started at http://127.21.201.66:34391/ using document root <none> and password file <none>
I20260812 01:01:38.051235 12157 fs_manager.cc:359] Metadata directory not provided
I20260812 01:01:38.051383 12157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:01:38.051792 12157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 01:01:38 dist-test-slave-7f01 krb5kdc[10855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496498, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for HTTP/127.21.201.84@KRBTEST.COM
I20260812 01:01:38.069465 12157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/data/instance:
uuid: "1f5cb71a9b6d4a6cbd272116e69cc67f"
format_stamp: "Formatted at 2026-08-12 01:01:38 on dist-test-slave-7f01"
server_key: "65ce5a635b0e4a7d3792bdb0b758c915"
server_key_iv: "85205ac4ca6c74cf58e1a4a6ec5f7b9b"
server_key_version: "kuduclusterkey@0"
I20260812 01:01:38.070734 12157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance:
uuid: "1f5cb71a9b6d4a6cbd272116e69cc67f"
format_stamp: "Formatted at 2026-08-12 01:01:38 on dist-test-slave-7f01"
server_key: "65ce5a635b0e4a7d3792bdb0b758c915"
server_key_iv: "85205ac4ca6c74cf58e1a4a6ec5f7b9b"
server_key_version: "kuduclusterkey@0"
I20260812 01:01:38.077939 12157 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260812 01:01:38.096879 12194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:38.098692 12157 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 01:01:38.098923 12157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "1f5cb71a9b6d4a6cbd272116e69cc67f"
format_stamp: "Formatted at 2026-08-12 01:01:38 on dist-test-slave-7f01"
server_key: "65ce5a635b0e4a7d3792bdb0b758c915"
server_key_iv: "85205ac4ca6c74cf58e1a4a6ec5f7b9b"
server_key_version: "kuduclusterkey@0"
I20260812 01:01:38.099257 12157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:01:38.123678 12157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:01:38.128037 12157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:01:38.128355 12157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:01:38.129550 12157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:01:38.131072 12157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:01:38.131176 12157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:38.131325 12157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:01:38.131398 12157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:38.173008 12157 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:35365
I20260812 01:01:38.173048 12400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:35365 every 8 connection(s)
I20260812 01:01:38.174343 12157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 01:01:38.181787 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 12157
I20260812 01:01:38.181984 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance
I20260812 01:01:38.182575 22309 external_mini_cluster.cc:1428] Setting key 4fe47049712460571db8979a9d72e33f
Aug 12 01:01:38 dist-test-slave-7f01 krb5kdc[10855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496498, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:01:38.200054 12407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:38.177518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:53261 (local address 127.21.201.126:38675)
0812 01:01:38.178162 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:38.178171 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:38.179066 (+ 895us) server_negotiation.cc:407] Connection header received
0812 01:01:38.181058 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:38.181066 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:38.181170 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:38.181294 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:38.183184 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:38.184480 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:38.186933 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:38.187581 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:38.187708 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:38.192688 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:38.192696 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:38.192702 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:38.192743 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:38.196486 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:38.197209 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:38.197216 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:38.197220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:38.197390 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:38.197992 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:38.197997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:38.198001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:38.198380 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:38.198531 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:38.199302 (+ 771us) server_negotiation.cc:299] Negotiation successful
0812 01:01:38.199549 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":324,"threads_started":1}
I20260812 01:01:38.201054 12408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:38.178015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38675 (local address 127.21.201.66:53261)
0812 01:01:38.178881 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:38.178927 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:38.180796 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:38.181556 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:38.181571 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:38.182188 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:01:38.182979 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:38.183006 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:38.184657 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:38.184663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:38.186752 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:38.186784 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:38.187879 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:38.187889 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:38.188241 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:38.189347 (+ 1106us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:01:38.189389 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:01:38.192441 (+ 3052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:01:38.196640 (+ 4199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:38.196647 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:38.196662 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:38.197090 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:38.197531 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:38.197538 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:38.197543 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:38.197812 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:38.198515 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:01:38.198521 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:01:38.198860 (+ 339us) client_negotiation.cc:769] Sending connection context
0812 01:01:38.199202 (+ 342us) client_negotiation.cc:240] Negotiation successful
0812 01:01:38.199728 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":377,"threads_started":1}
I20260812 01:01:38.202561 12402 heartbeater.cc:347] Connected to a master server at 127.21.201.126:38675
I20260812 01:01:38.202900 12402 heartbeater.cc:464] Registering TS with master...
I20260812 01:01:38.203724 12402 heartbeater.cc:511] Master 127.21.201.126:38675 requested a full tablet report, sending...
I20260812 01:01:38.205374 11745 ts_manager.cc:194] Registered new tserver with Master: 1f5cb71a9b6d4a6cbd272116e69cc67f (127.21.201.66:35365)
I20260812 01:01:38.206411 11745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.66@KRBTEST.COM'} at 127.21.201.66:53261
WARNING: no policy specified for kudu/127.21.201.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
I20260812 01:01:38.255614 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.201.84:57545/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:0
--local_ip_for_outbound_sockets=127.21.201.67
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38675
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:44549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:01:38.412249 12145 heartbeater.cc:503] Master 127.21.201.126:38675 was elected leader, sending a full tablet report...
W20260812 01:01:38.599165 12414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:01:38.599439 12414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:01:38.599596 12414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:01:38.607012 12414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:01:38.607161 12414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:01:38.607333 12414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 01:01:38.617513 12414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:44549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.201.84:57545/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:01:38.618870 12414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:01:38.620257 12414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:01:38.633915 12432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:38.634014 12424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:38.634008 12429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:01:38.634753 12414 server_base.cc:1034] running on GCE node
I20260812 01:01:38.635341 12414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:01:38.636319 12414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:01:38.637603 12414 hybrid_clock.cc:648] HybridClock initialized: now 1786496498637558 us; error 46 us; skew 500 ppm
I20260812 01:01:38.638011 12414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:01:38 dist-test-slave-7f01 krb5kdc[10855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496498, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:01:38.641571 12414 init.cc:377] Logged in from keytab as kudu/127.21.201.67@KRBTEST.COM (short username kudu)
I20260812 01:01:38.643065 12414 webserver.cc:460] Webserver started at http://127.21.201.67:44711/ using document root <none> and password file <none>
I20260812 01:01:38.643860 12414 fs_manager.cc:359] Metadata directory not provided
I20260812 01:01:38.644016 12414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:01:38.644349 12414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 01:01:38 dist-test-slave-7f01 krb5kdc[10855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496498, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for HTTP/127.21.201.84@KRBTEST.COM
I20260812 01:01:38.660287 12414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/data/instance:
uuid: "b5ee1cd9a5e44f139e55591e25a61035"
format_stamp: "Formatted at 2026-08-12 01:01:38 on dist-test-slave-7f01"
server_key: "06ea19d23cd6d50af7efa012d4db7171"
server_key_iv: "4bf47f29c66c61d8841de4164fa6ab31"
server_key_version: "kuduclusterkey@0"
I20260812 01:01:38.661511 12414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance:
uuid: "b5ee1cd9a5e44f139e55591e25a61035"
format_stamp: "Formatted at 2026-08-12 01:01:38 on dist-test-slave-7f01"
server_key: "06ea19d23cd6d50af7efa012d4db7171"
server_key_iv: "4bf47f29c66c61d8841de4164fa6ab31"
server_key_version: "kuduclusterkey@0"
I20260812 01:01:38.667785 12414 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 01:01:38.684396 12447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:38.685930 12414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 01:01:38.686115 12414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "b5ee1cd9a5e44f139e55591e25a61035"
format_stamp: "Formatted at 2026-08-12 01:01:38 on dist-test-slave-7f01"
server_key: "06ea19d23cd6d50af7efa012d4db7171"
server_key_iv: "4bf47f29c66c61d8841de4164fa6ab31"
server_key_version: "kuduclusterkey@0"
I20260812 01:01:38.686358 12414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:01:38.699901 12414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:01:38.704167 12414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:01:38.704526 12414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:01:38.705567 12414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:01:38.706985 12414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:01:38.707060 12414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:38.707203 12414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:01:38.707270 12414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:38.743312 12414 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:35165
I20260812 01:01:38.743347 12649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:35165 every 8 connection(s)
I20260812 01:01:38.744640 12414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 01:01:38.752821 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 12414
I20260812 01:01:38.753043 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance
I20260812 01:01:38.753595 22309 external_mini_cluster.cc:1428] Setting key 2cc033f816fcff20ddc58a38fef15b5b
Aug 12 01:01:38 dist-test-slave-7f01 krb5kdc[10855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496498, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:01:38.766203 12654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:38.747369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:48821 (local address 127.21.201.126:38675)
0812 01:01:38.747919 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:38.747925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:38.748702 (+ 777us) server_negotiation.cc:407] Connection header received
0812 01:01:38.750611 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:38.750618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:38.750721 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:38.750841 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:38.752223 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:38.753411 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:38.755831 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:38.756534 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:38.756650 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:38.760501 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:38.760507 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:38.760510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:38.760551 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:38.763299 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:38.764123 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:38.764128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:38.764130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:38.764261 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:38.764711 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:38.764715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:38.764717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:38.764992 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:38.765132 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:38.765775 (+ 643us) server_negotiation.cc:299] Negotiation successful
0812 01:01:38.765954 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":240,"threads_started":1}
I20260812 01:01:38.767215 12656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:38.747745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38675 (local address 127.21.201.67:48821)
0812 01:01:38.748530 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:38.748574 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:38.750335 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:38.751034 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:38.751041 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:38.751447 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:01:38.752060 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:38.752077 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:38.753611 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:38.753617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:38.755626 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:38.755642 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:38.756943 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:38.756950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:38.757248 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:38.758105 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:01:38.758129 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:01:38.760259 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:01:38.763441 (+ 3182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:38.763448 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:38.763460 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:38.763977 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:38.764401 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:38.764405 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:38.764407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:38.764596 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:38.765139 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:01:38.765146 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:01:38.765449 (+ 303us) client_negotiation.cc:769] Sending connection context
0812 01:01:38.765677 (+ 228us) client_negotiation.cc:240] Negotiation successful
0812 01:01:38.766058 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":321,"threads_started":1}
I20260812 01:01:38.768731 12652 heartbeater.cc:347] Connected to a master server at 127.21.201.126:38675
I20260812 01:01:38.769016 12652 heartbeater.cc:464] Registering TS with master...
I20260812 01:01:38.769843 12652 heartbeater.cc:511] Master 127.21.201.126:38675 requested a full tablet report, sending...
I20260812 01:01:38.771389 11745 ts_manager.cc:194] Registered new tserver with Master: b5ee1cd9a5e44f139e55591e25a61035 (127.21.201.67:35165)
I20260812 01:01:38.772207 11745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.67@KRBTEST.COM'} at 127.21.201.67:48821
I20260812 01:01:38.782904 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:01:38.784770 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 11904
Aug 12 01:01:38 dist-test-slave-7f01 krb5kdc[10855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.65@KRBTEST.COM
I20260812 01:01:38.802416 12658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:38.785746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35332 (local address 127.21.201.65:43425)
0812 01:01:38.786351 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:38.786358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:38.786383 (+ 25us) server_negotiation.cc:407] Connection header received
0812 01:01:38.786457 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:38.786463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:38.786647 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:38.786789 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:38.788332 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:38.789400 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:38.791273 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:38.791876 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:38.791976 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:38.796594 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:38.796605 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:38.796618 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:38.796648 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:38.799355 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:38.800173 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:38.800180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:38.800182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:38.800356 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:38.800815 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:38.800819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:38.800821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:38.801142 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:38.801483 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:38.801757 (+ 274us) server_negotiation.cc:299] Negotiation successful
0812 01:01:38.802076 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":303,"threads_started":1}
I20260812 01:01:38.803923 12135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:01:38.887494 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 11904
I20260812 01:01:38.913024 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 12157
Aug 12 01:01:38 dist-test-slave-7f01 krb5kdc[10855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.66@KRBTEST.COM
I20260812 01:01:38.930071 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:38.913972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48408 (local address 127.21.201.66:35365)
0812 01:01:38.914581 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:38.914588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:38.914615 (+ 27us) server_negotiation.cc:407] Connection header received
0812 01:01:38.914679 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:38.914685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:38.914908 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:38.915037 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:38.916385 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:38.917451 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:38.918844 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:38.919376 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:38.919481 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:38.924042 (+ 4561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:38.924054 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:38.924067 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:38.924098 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:38.927001 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:38.927874 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:38.927881 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:38.927883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:38.928019 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:38.928423 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:38.928429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:38.928432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:38.928778 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:38.929133 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:38.929427 (+ 294us) server_negotiation.cc:299] Negotiation successful
0812 01:01:38.929699 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":331,"threads_started":1}
I20260812 01:01:38.931310 12395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:01:39.014143 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 12157
I20260812 01:01:39.043105 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 12414
Aug 12 01:01:39 dist-test-slave-7f01 krb5kdc[10855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.67@KRBTEST.COM
I20260812 01:01:39.060127 12662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:39.044157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53928 (local address 127.21.201.67:35165)
0812 01:01:39.044750 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:39.044757 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:39.044788 (+ 31us) server_negotiation.cc:407] Connection header received
0812 01:01:39.044852 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:39.044858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:39.045065 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:39.045255 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:39.046668 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:39.047817 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:39.049203 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:39.049778 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:39.049923 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:39.054610 (+ 4687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:39.054622 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:39.054637 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:39.054673 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:39.057590 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:39.058278 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:39.058284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:39.058287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:39.058420 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:39.058780 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:39.058783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:39.058785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:39.059106 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:39.059344 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:39.059591 (+ 247us) server_negotiation.cc:299] Negotiation successful
0812 01:01:39.059840 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":287,"threads_started":1}
I20260812 01:01:39.061458 12643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:01:39.143415 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 12414
I20260812 01:01:39.163847 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 11683
I20260812 01:01:39.165187 11805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:01:39.276683 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 11683
I20260812 01:01:39.313146 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:38675
--webserver_interface=127.21.201.126
--webserver_port=46843
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:44549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:38675
--ranger_config_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.201.84:57545/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.201.84:57545/kms with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:01:39.649469 12670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:01:39.649829 12670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:01:39.649998 12670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:01:39.657292 12670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:01:39.657429 12670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:01:39.657536 12670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:01:39.657634 12670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:01:39.657722 12670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:01:39.667754 12670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:44549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.201.84:57545/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:38675
--ranger_config_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:38675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=46843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:01:39.669049 12670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:01:39.670368 12670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:01:39.682173 12684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:39.682304 12685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:39.683025 12687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:01:39.683526 12670 server_base.cc:1034] running on GCE node
I20260812 01:01:39.684388 12670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:01:39.685802 12670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:01:39.687007 12670 hybrid_clock.cc:648] HybridClock initialized: now 1786496499686960 us; error 46 us; skew 500 ppm
I20260812 01:01:39.687366 12670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:01:39 dist-test-slave-7f01 krb5kdc[10855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496499, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:01:39.690620 12670 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 01:01:39.692173 12670 webserver.cc:460] Webserver started at http://127.21.201.126:46843/ using document root <none> and password file <none>
I20260812 01:01:39.692776 12670 fs_manager.cc:359] Metadata directory not provided
I20260812 01:01:39.692895 12670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 01:01:39 dist-test-slave-7f01 krb5kdc[10855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496499, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for HTTP/127.21.201.84@KRBTEST.COM
I20260812 01:01:39.745904 12670 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 01:01:39.749112 12700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:39.750476 12670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 01:01:39.750680 12670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "7e0928e827fe43ae9ae7af7ca7fbb828"
format_stamp: "Formatted at 2026-08-12 01:01:35 on dist-test-slave-7f01"
server_key: "f47fc8f04eea86bceb3e35502104f2cf"
server_key_iv: "c74fd1ee42ecdecba0236932e666aa59"
server_key_version: "kuduclusterkey@0"
I20260812 01:01:39.751688 12670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:01:39.769481 12670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:01:39.773408 12670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:01:39.773713 12670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:01:39.793403 12670 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:38675
I20260812 01:01:39.793435 12788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:38675 every 8 connection(s)
I20260812 01:01:39.794909 12670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:01:39.796238 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 12670
I20260812 01:01:39.797123 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.201.84:57545/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:43425
--local_ip_for_outbound_sockets=127.21.201.65
--tserver_master_addrs=127.21.201.126:38675
--webserver_port=38401
--webserver_interface=127.21.201.65
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:44549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.201.84:57545/kms with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:01:39.802302 12790 sys_catalog.cc:263] Verifying existing consensus state
I20260812 01:01:39.805629 12790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828: Bootstrap starting.
I20260812 01:01:39.825265 12790 log.cc:824] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828: Log is configured to *not* fsync() on all Append() calls
I20260812 01:01:39.834514 12790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 01:01:39.835078 12790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828: Bootstrap complete.
I20260812 01:01:39.844182 12790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0928e827fe43ae9ae7af7ca7fbb828" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38675 } }
I20260812 01:01:39.845045 12790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e0928e827fe43ae9ae7af7ca7fbb828, State: Initialized, Role: FOLLOWER
I20260812 01:01:39.845623 12790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0928e827fe43ae9ae7af7ca7fbb828" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38675 } }
I20260812 01:01:39.845876 12790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:01:39.846007 12790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:01:39.846191 12790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [term 1 FOLLOWER]: Advancing to term 2
I20260812 01:01:39.850045 12790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0928e827fe43ae9ae7af7ca7fbb828" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38675 } }
I20260812 01:01:39.850513 12790 leader_election.cc:302] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e0928e827fe43ae9ae7af7ca7fbb828; no voters:
I20260812 01:01:39.851174 12790 leader_election.cc:288] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 01:01:39.851330 12796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 01:01:39.852773 12796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [term 2 LEADER]: Becoming Leader. State: Replica: 7e0928e827fe43ae9ae7af7ca7fbb828, State: Running, Role: LEADER
I20260812 01:01:39.853538 12790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:01:39.853389 12796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0928e827fe43ae9ae7af7ca7fbb828" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38675 } }
I20260812 01:01:39.856307 12797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7e0928e827fe43ae9ae7af7ca7fbb828" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0928e827fe43ae9ae7af7ca7fbb828" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38675 } } }
I20260812 01:01:39.856701 12797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [sys.catalog]: This master's current role is: LEADER
I20260812 01:01:39.856436 12798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e0928e827fe43ae9ae7af7ca7fbb828. Latest consensus state: current_term: 2 leader_uuid: "7e0928e827fe43ae9ae7af7ca7fbb828" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0928e827fe43ae9ae7af7ca7fbb828" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38675 } } }
I20260812 01:01:39.856829 12798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828 [sys.catalog]: This master's current role is: LEADER
I20260812 01:01:39.862937 12804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:01:39.869952 12804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:01:39.871850 12804 catalog_manager.cc:1259] Loaded cluster ID: 00368148a6d44611b570a9a169145fc2
I20260812 01:01:39.872028 12804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:01:39.877719 12804 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:01:39.880674 12804 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828: Loaded TSK: 0
I20260812 01:01:39.882840 12804 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:01:39.886006 12790 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260812 01:01:40.260195 12792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:01:40.260572 12792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:01:40.260738 12792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:01:40.272699 12792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:01:40.272861 12792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:01:40.273069 12792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 01:01:40.288957 12792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:44549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.201.84:57545/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:43425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=38401
--tserver_master_addrs=127.21.201.126:38675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:01:40.290583 12792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:01:40.292294 12792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:01:40.347856 12850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:40.352795 12851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:01:40.376061 12792 server_base.cc:1034] running on GCE node
W20260812 01:01:40.377467 12853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:01:40.378444 12792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:01:40.386056 12792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:01:40.387506 12792 hybrid_clock.cc:648] HybridClock initialized: now 1786496500387391 us; error 113 us; skew 500 ppm
I20260812 01:01:40.388018 12792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:01:40 dist-test-slave-7f01 krb5kdc[10855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496500, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:01:40.392619 12792 init.cc:377] Logged in from keytab as kudu/127.21.201.65@KRBTEST.COM (short username kudu)
I20260812 01:01:40.419282 12792 webserver.cc:460] Webserver started at http://127.21.201.65:38401/ using document root <none> and password file <none>
I20260812 01:01:40.420362 12792 fs_manager.cc:359] Metadata directory not provided
I20260812 01:01:40.420547 12792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 01:01:40 dist-test-slave-7f01 krb5kdc[10855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496500, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for HTTP/127.21.201.84@KRBTEST.COM
I20260812 01:01:40.471390 12792 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.004s sys 0.002s
I20260812 01:01:40.476445 12880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:40.478211 12792 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20260812 01:01:40.478489 12792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "471561b216ca438bb4464871c8a516b3"
format_stamp: "Formatted at 2026-08-12 01:01:37 on dist-test-slave-7f01"
server_key: "58fcdc07cf837fbedd91fc9952dd3a9d"
server_key_iv: "c93b84d9e4755f9f30cf558a6136d022"
server_key_version: "kuduclusterkey@0"
I20260812 01:01:40.479598 12792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:01:40.503249 12792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:01:40.508543 12792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:01:40.508865 12792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:01:40.510228 12792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:01:40.512210 12792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:01:40.512355 12792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:40.512485 12792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:01:40.512585 12792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:40.572141 12792 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:43425
I20260812 01:01:40.572201 13068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:43425 every 8 connection(s)
I20260812 01:01:40.573645 12792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 01:01:40.576539 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 12792
I20260812 01:01:40.577224 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.201.84:57545/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:35365
--local_ip_for_outbound_sockets=127.21.201.66
--tserver_master_addrs=127.21.201.126:38675
--webserver_port=34391
--webserver_interface=127.21.201.66
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:44549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.201.84:57545/kms with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 01:01:40 dist-test-slave-7f01 krb5kdc[10855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496500, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:01:40.612469 13072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:40.577186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38675 (local address 127.21.201.65:47885)
0812 01:01:40.578211 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:40.578263 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:40.581028 (+ 2765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:40.586154 (+ 5126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:40.586168 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:40.586879 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:01:40.587717 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:40.587774 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:40.590706 (+ 2932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:40.590715 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:40.592992 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:40.593012 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:40.594666 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:40.594679 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:40.595033 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:40.595893 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:01:40.595930 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:01:40.599059 (+ 3129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:01:40.607097 (+ 8038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:40.607108 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:40.607126 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:40.607815 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:40.608421 (+ 606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:40.608426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:40.608430 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:40.608690 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:40.609700 (+ 1010us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:01:40.609710 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:01:40.610124 (+ 414us) client_negotiation.cc:769] Sending connection context
0812 01:01:40.610427 (+ 303us) client_negotiation.cc:240] Negotiation successful
0812 01:01:40.611021 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":453,"threads_started":1}
I20260812 01:01:40.615450 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:40.579827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:47885 (local address 127.21.201.126:38675)
0812 01:01:40.584367 (+ 4540us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:40.584389 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:40.584444 (+ 55us) server_negotiation.cc:407] Connection header received
0812 01:01:40.585045 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:40.585070 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:40.585548 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:40.587935 (+ 2387us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:40.588423 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:40.590497 (+ 2074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:40.593219 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:40.594378 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:40.595547 (+ 1169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:40.599435 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:40.599447 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:40.599467 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:40.599529 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:40.606876 (+ 7347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:40.608030 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:40.608039 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:40.608048 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:40.608269 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:40.608856 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:40.608861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:40.608864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:40.609526 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:40.613412 (+ 3886us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:40.613705 (+ 293us) server_negotiation.cc:299] Negotiation successful
0812 01:01:40.614246 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4222,"thread_start_us":6000,"threads_started":1}
I20260812 01:01:40.618883 13069 heartbeater.cc:347] Connected to a master server at 127.21.201.126:38675
I20260812 01:01:40.619234 13069 heartbeater.cc:464] Registering TS with master...
I20260812 01:01:40.620226 13069 heartbeater.cc:511] Master 127.21.201.126:38675 requested a full tablet report, sending...
W20260812 01:01:40.623783 13069 heartbeater.cc:644] Failed to heartbeat to 127.21.201.126:38675 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 01:01:40.624042 13069 heartbeater.cc:464] Registering TS with master...
I20260812 01:01:40.624339 13069 heartbeater.cc:511] Master 127.21.201.126:38675 requested a full tablet report, sending...
I20260812 01:01:40.625583 13069 heartbeater.cc:464] Registering TS with master...
I20260812 01:01:40.625849 13069 heartbeater.cc:511] Master 127.21.201.126:38675 requested a full tablet report, sending...
W20260812 01:01:40.626928 13069 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 01:01:41.029418 13075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:01:41.029752 13075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:01:41.029934 13075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:01:41.037200 13075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:01:41.037405 13075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:01:41.037600 13075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 01:01:41.053054 13075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:44549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.201.84:57545/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:35365
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=34391
--tserver_master_addrs=127.21.201.126:38675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:01:41.054916 13075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:01:41.056710 13075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:01:41.091003 13105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:41.091392 13109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:41.108914 13115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:01:41.113488 13075 server_base.cc:1034] running on GCE node
I20260812 01:01:41.114423 13075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:01:41.115608 13075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:01:41.116911 13075 hybrid_clock.cc:648] HybridClock initialized: now 1786496501116824 us; error 59 us; skew 500 ppm
I20260812 01:01:41.117327 13075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:01:41 dist-test-slave-7f01 krb5kdc[10855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496501, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:01:41.120880 13075 init.cc:377] Logged in from keytab as kudu/127.21.201.66@KRBTEST.COM (short username kudu)
I20260812 01:01:41.122848 13075 webserver.cc:460] Webserver started at http://127.21.201.66:34391/ using document root <none> and password file <none>
I20260812 01:01:41.123629 13075 fs_manager.cc:359] Metadata directory not provided
I20260812 01:01:41.123821 13075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 01:01:41 dist-test-slave-7f01 krb5kdc[10855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496501, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for HTTP/127.21.201.84@KRBTEST.COM
I20260812 01:01:41.147981 13075 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 01:01:41.151031 13130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:41.152848 13075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:01:41.153044 13075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "1f5cb71a9b6d4a6cbd272116e69cc67f"
format_stamp: "Formatted at 2026-08-12 01:01:38 on dist-test-slave-7f01"
server_key: "65ce5a635b0e4a7d3792bdb0b758c915"
server_key_iv: "85205ac4ca6c74cf58e1a4a6ec5f7b9b"
server_key_version: "kuduclusterkey@0"
I20260812 01:01:41.154533 13075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 12 01:01:41 dist-test-slave-7f01 krb5kdc[10855](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786496501, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:01:41.221359 13075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:01:41.226091 13075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:01:41.226477 13075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:01:41.228019 13075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:01:41.229586 13075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:01:41.229720 13075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:41.229847 13075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:01:41.229938 13075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:41.290548 13075 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:35365
I20260812 01:01:41.291142 13312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:35365 every 8 connection(s)
I20260812 01:01:41.292377 13075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 01:01:41.294899 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 13075
I20260812 01:01:41.295727 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.201.84:57545/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:35165
--local_ip_for_outbound_sockets=127.21.201.67
--tserver_master_addrs=127.21.201.126:38675
--webserver_port=44711
--webserver_interface=127.21.201.67
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:44549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.201.84:57545/kms with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 01:01:41 dist-test-slave-7f01 krb5kdc[10855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496501, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:01:41.324467 13317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:41.297433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:60635 (local address 127.21.201.126:38675)
0812 01:01:41.303568 (+ 6135us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:41.303574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:41.303598 (+ 24us) server_negotiation.cc:407] Connection header received
0812 01:01:41.303669 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:41.303675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:41.303788 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:41.303929 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:41.306163 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:41.307374 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:41.311077 (+ 3703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:41.311788 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:41.311908 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:41.316493 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:41.316500 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:41.316505 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:41.316552 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:41.320325 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:41.321339 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:41.321345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:41.321349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:41.321523 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:41.322127 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:41.322133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:41.322136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:41.322575 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:41.322708 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:41.323783 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0812 01:01:41.324041 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5985,"thread_start_us":6167,"threads_started":1}
I20260812 01:01:41.327246 13316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:41.296504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38675 (local address 127.21.201.66:60635)
0812 01:01:41.298041 (+ 1537us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:41.298104 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:41.301274 (+ 3170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:41.304439 (+ 3165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:41.304451 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:41.305148 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:01:41.305938 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:41.305965 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:41.307588 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:41.307594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:41.309671 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:41.309697 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:41.312276 (+ 2579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:41.312289 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:41.312612 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:41.313467 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:01:41.313509 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:01:41.316191 (+ 2682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:01:41.320522 (+ 4331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:41.320532 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:41.320549 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:41.321176 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:41.321705 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:41.321710 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:41.321713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:41.321980 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:41.322820 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:01:41.322829 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:01:41.323277 (+ 448us) client_negotiation.cc:769] Sending connection context
0812 01:01:41.325093 (+ 1816us) client_negotiation.cc:240] Negotiation successful
0812 01:01:41.325714 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1228,"thread_start_us":960,"threads_started":1}
I20260812 01:01:41.329670 13313 heartbeater.cc:347] Connected to a master server at 127.21.201.126:38675
I20260812 01:01:41.330019 13313 heartbeater.cc:464] Registering TS with master...
I20260812 01:01:41.330968 13313 heartbeater.cc:511] Master 127.21.201.126:38675 requested a full tablet report, sending...
I20260812 01:01:41.333248 12736 ts_manager.cc:194] Registered new tserver with Master: 1f5cb71a9b6d4a6cbd272116e69cc67f (127.21.201.66:35365)
I20260812 01:01:41.335467 12736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.66@KRBTEST.COM'} at 127.21.201.66:60635
I20260812 01:01:41.627123 13069 heartbeater.cc:464] Registering TS with master...
I20260812 01:01:41.627635 13069 heartbeater.cc:511] Master 127.21.201.126:38675 requested a full tablet report, sending...
I20260812 01:01:41.629181 12736 ts_manager.cc:194] Registered new tserver with Master: 471561b216ca438bb4464871c8a516b3 (127.21.201.65:43425)
I20260812 01:01:41.630436 12736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.65@KRBTEST.COM'} at 127.21.201.65:47885
Aug 12 01:01:41 dist-test-slave-7f01 krb5kdc[10855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496501, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for HTTP/127.21.201.84@KRBTEST.COM
W20260812 01:01:41.767518 13318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:01:41.768049 13318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:01:41.768262 13318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:01:41.780035 13318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:01:41.780282 13318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:01:41.780525 13318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 01:01:41.796558 13318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:44549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.201.84:57545/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:35165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=44711
--tserver_master_addrs=127.21.201.126:38675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:01:41.798527 13318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:01:41.800408 13318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:01:41.837224 13331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:01:41.840790 13318 server_base.cc:1034] running on GCE node
W20260812 01:01:41.838284 13332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:41.844071 13337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:01:41.844931 13318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:01:41.848086 13318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:01:41.849352 13318 hybrid_clock.cc:648] HybridClock initialized: now 1786496501849286 us; error 62 us; skew 500 ppm
I20260812 01:01:41.849704 13318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:01:41 dist-test-slave-7f01 krb5kdc[10855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496501, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:01:41.853200 13318 init.cc:377] Logged in from keytab as kudu/127.21.201.67@KRBTEST.COM (short username kudu)
I20260812 01:01:41.857184 13318 webserver.cc:460] Webserver started at http://127.21.201.67:44711/ using document root <none> and password file <none>
I20260812 01:01:41.857908 13318 fs_manager.cc:359] Metadata directory not provided
I20260812 01:01:41.858080 13318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 01:01:41 dist-test-slave-7f01 krb5kdc[10855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496501, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for HTTP/127.21.201.84@KRBTEST.COM
I20260812 01:01:41.898350 13318 fs_manager.cc:709] Time spent opening directory manager: real 0.021s user 0.005s sys 0.012s
I20260812 01:01:41.901733 13350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:41.903770 13318 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 01:01:41.904017 13318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "b5ee1cd9a5e44f139e55591e25a61035"
format_stamp: "Formatted at 2026-08-12 01:01:38 on dist-test-slave-7f01"
server_key: "06ea19d23cd6d50af7efa012d4db7171"
server_key_iv: "4bf47f29c66c61d8841de4164fa6ab31"
server_key_version: "kuduclusterkey@0"
I20260812 01:01:41.905081 13318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:01:41.918068 13318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:01:41.922225 13318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:01:41.922578 13318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:01:41.923657 13318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:01:41.925204 13318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:01:41.925328 13318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:41.925469 13318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:01:41.925573 13318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:41.990988 13318 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:35165
I20260812 01:01:41.991067 13546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:35165 every 8 connection(s)
I20260812 01:01:41.993077 13318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 01:01:41.994045 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 13318
I20260812 01:01:42.051867 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:41.996302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53378 (local address 127.21.201.126:38675)
0812 01:01:42.022677 (+ 26375us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:42.022682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:42.022701 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:01:42.022772 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:42.022776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:42.022838 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:42.022944 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:42.028634 (+ 5690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.029868 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.035276 (+ 5408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.036062 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.036186 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:42.041951 (+ 5765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:42.041956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:42.041961 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:42.042024 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:42.047552 (+ 5528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:42.048549 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:42.048555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:42.048559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:42.048715 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:42.049233 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:42.049238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:42.049241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:42.049602 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:42.049858 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:42.051191 (+ 1333us) server_negotiation.cc:299] Negotiation successful
0812 01:01:42.051444 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26263,"thread_start_us":26144,"threads_started":1}
Aug 12 01:01:42 dist-test-slave-7f01 krb5kdc[10855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496501, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
Aug 12 01:01:42 dist-test-slave-7f01 krb5kdc[10855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496501, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for HTTP/127.21.201.84@KRBTEST.COM
I20260812 01:01:42.070200 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:42.032595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:44427 (local address 127.21.201.126:38675)
0812 01:01:42.041774 (+ 9179us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:42.041779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:42.041799 (+ 20us) server_negotiation.cc:407] Connection header received
0812 01:01:42.042690 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:42.042697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:42.042791 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:42.042919 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:42.045886 (+ 2967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.048090 (+ 2204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.056052 (+ 7962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.056591 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.056731 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:42.061073 (+ 4342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:42.061079 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:42.061082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:42.061117 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:42.064371 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:42.066067 (+ 1696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:42.066073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:42.066076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:42.066257 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:42.066812 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:42.066817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:42.066821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:42.067189 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:42.067333 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:42.069569 (+ 2236us) server_negotiation.cc:299] Negotiation successful
0812 01:01:42.069789 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9046,"thread_start_us":8857,"threads_started":1}
I20260812 01:01:42.071640 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:42.033263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38675 (local address 127.21.201.67:44427)
0812 01:01:42.039871 (+ 6608us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:42.039928 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:42.042338 (+ 2410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:42.043203 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:42.043215 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:42.044841 (+ 1626us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:01:42.045630 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:42.045657 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.048938 (+ 3281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.048944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:42.055891 (+ 6947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:42.055906 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.056844 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.056852 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:42.057114 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:42.057893 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:01:42.058041 (+ 148us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:01:42.060783 (+ 2742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:01:42.064546 (+ 3763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:42.064555 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:42.064573 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:42.065908 (+ 1335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:42.066394 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:42.066399 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:42.066403 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:42.066646 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:42.067411 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:01:42.067420 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:01:42.067901 (+ 481us) client_negotiation.cc:769] Sending connection context
0812 01:01:42.068229 (+ 328us) client_negotiation.cc:240] Negotiation successful
0812 01:01:42.068816 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6266,"thread_start_us":8181,"threads_started":1}
I20260812 01:01:42.073091 13547 heartbeater.cc:347] Connected to a master server at 127.21.201.126:38675
I20260812 01:01:42.073467 13547 heartbeater.cc:464] Registering TS with master...
I20260812 01:01:42.074375 13547 heartbeater.cc:511] Master 127.21.201.126:38675 requested a full tablet report, sending...
I20260812 01:01:42.076107 12736 ts_manager.cc:194] Registered new tserver with Master: b5ee1cd9a5e44f139e55591e25a61035 (127.21.201.67:35165)
I20260812 01:01:42.077155 12736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.67@KRBTEST.COM'} at 127.21.201.67:44427
I20260812 01:01:42.080941 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:01:42.119876 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:42.099395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53388 (local address 127.21.201.126:38675)
0812 01:01:42.099613 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:42.099622 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:42.100078 (+ 456us) server_negotiation.cc:407] Connection header received
0812 01:01:42.100323 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:42.100329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:42.100427 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:42.100510 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:42.104389 (+ 3879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.105501 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.107300 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.107824 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.107973 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:42.111775 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:42.111784 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:42.111789 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:42.111841 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:42.115643 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:42.116666 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:42.116672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:42.116676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:42.116839 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:42.117351 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:42.117356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:42.117360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:42.117749 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:42.118016 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:42.119249 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0812 01:01:42.119491 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 01:01:42.125793 12736 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:01:42.136565 12736 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:01:42.201457 13581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:42.167264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:43425 (local address 127.0.0.1:35348)
0812 01:01:42.180402 (+ 13138us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:42.180450 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:42.180668 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:42.190748 (+ 10080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:42.190755 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:42.190812 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:01:42.191276 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:42.191295 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.194730 (+ 3435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.194736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:42.197293 (+ 2557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:42.197311 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.200484 (+ 3173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.200491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:42.200705 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:42.200762 (+ 57us) client_negotiation.cc:769] Sending connection context
0812 01:01:42.200883 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 01:01:42.201072 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12816,"thread_start_us":22981,"threads_started":1}
I20260812 01:01:42.204248 13580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:42.167601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35348 (local address 127.21.201.65:43425)
0812 01:01:42.189886 (+ 22285us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:42.189896 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:42.189926 (+ 30us) server_negotiation.cc:407] Connection header received
0812 01:01:42.190034 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:42.190043 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:42.190272 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:42.190730 (+ 458us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:01:42.191537 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.194560 (+ 3023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.198656 (+ 4096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.200230 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.202202 (+ 1972us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:42.203619 (+ 1417us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:42.203694 (+ 75us) server_negotiation.cc:299] Negotiation successful
0812 01:01:42.203902 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22079,"thread_start_us":22828,"threads_started":1}
I20260812 01:01:42.207948 13584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:42.173169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53932 (local address 127.21.201.67:35165)
0812 01:01:42.181356 (+ 8187us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:42.181377 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:42.188895 (+ 7518us) server_negotiation.cc:407] Connection header received
0812 01:01:42.189109 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:42.189118 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:42.189402 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:42.189633 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:01:42.200477 (+ 10844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.202069 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.204886 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.206429 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.206530 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:42.206735 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:42.207416 (+ 681us) server_negotiation.cc:299] Negotiation successful
0812 01:01:42.207571 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7990,"thread_start_us":9945,"threads_started":1}
I20260812 01:01:42.208860 13585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:42.172743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:35165 (local address 127.0.0.1:53932)
0812 01:01:42.188758 (+ 16015us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:42.188778 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:42.188931 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:42.199837 (+ 10906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:42.199844 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:42.199878 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:01:42.200221 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:42.200233 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.202435 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.202440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:42.204683 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:42.204700 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.206967 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.206975 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:42.207157 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:42.207199 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 01:01:42.208327 (+ 1128us) client_negotiation.cc:240] Negotiation successful
0812 01:01:42.208466 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15889,"mutex_wait_us":4,"thread_start_us":7629,"threads_started":1}
I20260812 01:01:42.209328 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 61820fdd49584758ab16f32e7a80cfed (DEFAULT_TABLE table=test-table [id=b3e8f19d360a4eae9cbaf332099179ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:01:42.213614 13440 tablet_service.cc:1467] Processing CreateTablet for tablet 61820fdd49584758ab16f32e7a80cfed (DEFAULT_TABLE table=test-table [id=b3e8f19d360a4eae9cbaf332099179ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:01:42.215142 13440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61820fdd49584758ab16f32e7a80cfed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:01:42.216091 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61820fdd49584758ab16f32e7a80cfed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:01:42.233129 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:42.181672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48418 (local address 127.21.201.66:35365)
0812 01:01:42.183833 (+ 2161us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:42.183842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:42.190138 (+ 6296us) server_negotiation.cc:407] Connection header received
0812 01:01:42.190313 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:42.190321 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:42.190519 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:42.190733 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:01:42.199766 (+ 9033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.201412 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.204021 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.205627 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.205736 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:42.205941 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:42.232537 (+ 26596us) server_negotiation.cc:299] Negotiation successful
0812 01:01:42.232761 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1983,"thread_start_us":1752,"threads_started":1}
I20260812 01:01:42.234155 13579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:42.181360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:35365 (local address 127.0.0.1:48418)
0812 01:01:42.190025 (+ 8665us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:42.190042 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:42.190162 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:42.199080 (+ 8918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:42.199086 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:42.199116 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:01:42.199537 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:42.199549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.201611 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.201617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:42.203841 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:42.203857 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.232022 (+ 28165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.232029 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:42.232251 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:42.232292 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 01:01:42.233625 (+ 1333us) client_negotiation.cc:240] Negotiation successful
0812 01:01:42.233812 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8557,"thread_start_us":7184,"threads_started":1}
I20260812 01:01:42.238571 13209 tablet_service.cc:1467] Processing CreateTablet for tablet 61820fdd49584758ab16f32e7a80cfed (DEFAULT_TABLE table=test-table [id=b3e8f19d360a4eae9cbaf332099179ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:01:42.240458 13209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61820fdd49584758ab16f32e7a80cfed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:01:42.271346 13588 tablet_bootstrap.cc:492] T 61820fdd49584758ab16f32e7a80cfed P 1f5cb71a9b6d4a6cbd272116e69cc67f: Bootstrap starting.
I20260812 01:01:42.274832 13588 tablet_bootstrap.cc:654] T 61820fdd49584758ab16f32e7a80cfed P 1f5cb71a9b6d4a6cbd272116e69cc67f: Neither blocks nor log segments found. Creating new log.
I20260812 01:01:42.275990 13588 log.cc:824] T 61820fdd49584758ab16f32e7a80cfed P 1f5cb71a9b6d4a6cbd272116e69cc67f: Log is configured to *not* fsync() on all Append() calls
I20260812 01:01:42.277091 13590 tablet_bootstrap.cc:492] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3: Bootstrap starting.
I20260812 01:01:42.279997 13590 tablet_bootstrap.cc:654] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3: Neither blocks nor log segments found. Creating new log.
I20260812 01:01:42.281361 13590 log.cc:824] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3: Log is configured to *not* fsync() on all Append() calls
I20260812 01:01:42.282366 13591 tablet_bootstrap.cc:492] T 61820fdd49584758ab16f32e7a80cfed P b5ee1cd9a5e44f139e55591e25a61035: Bootstrap starting.
I20260812 01:01:42.286149 13588 tablet_bootstrap.cc:492] T 61820fdd49584758ab16f32e7a80cfed P 1f5cb71a9b6d4a6cbd272116e69cc67f: No bootstrap required, opened a new log
I20260812 01:01:42.286434 13588 ts_tablet_manager.cc:1397] T 61820fdd49584758ab16f32e7a80cfed P 1f5cb71a9b6d4a6cbd272116e69cc67f: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.006s
I20260812 01:01:42.288213 13590 tablet_bootstrap.cc:492] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3: No bootstrap required, opened a new log
I20260812 01:01:42.288571 13590 ts_tablet_manager.cc:1397] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20260812 01:01:42.289921 13591 tablet_bootstrap.cc:654] T 61820fdd49584758ab16f32e7a80cfed P b5ee1cd9a5e44f139e55591e25a61035: Neither blocks nor log segments found. Creating new log.
I20260812 01:01:42.291218 13591 log.cc:824] T 61820fdd49584758ab16f32e7a80cfed P b5ee1cd9a5e44f139e55591e25a61035: Log is configured to *not* fsync() on all Append() calls
I20260812 01:01:42.305248 13590 raft_consensus.cc:348] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471561b216ca438bb4464871c8a516b3" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43425 } } peers { permanent_uuid: "b5ee1cd9a5e44f139e55591e25a61035" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35165 } } peers { permanent_uuid: "1f5cb71a9b6d4a6cbd272116e69cc67f" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35365 } }
I20260812 01:01:42.311049 13588 raft_consensus.cc:348] T 61820fdd49584758ab16f32e7a80cfed P 1f5cb71a9b6d4a6cbd272116e69cc67f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471561b216ca438bb4464871c8a516b3" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43425 } } peers { permanent_uuid: "b5ee1cd9a5e44f139e55591e25a61035" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35165 } } peers { permanent_uuid: "1f5cb71a9b6d4a6cbd272116e69cc67f" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35365 } }
I20260812 01:01:42.312621 13588 raft_consensus.cc:374] T 61820fdd49584758ab16f32e7a80cfed P 1f5cb71a9b6d4a6cbd272116e69cc67f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:01:42.312772 13588 raft_consensus.cc:729] T 61820fdd49584758ab16f32e7a80cfed P 1f5cb71a9b6d4a6cbd272116e69cc67f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f5cb71a9b6d4a6cbd272116e69cc67f, State: Initialized, Role: FOLLOWER
I20260812 01:01:42.313489 13588 consensus_queue.cc:260] T 61820fdd49584758ab16f32e7a80cfed P 1f5cb71a9b6d4a6cbd272116e69cc67f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471561b216ca438bb4464871c8a516b3" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43425 } } peers { permanent_uuid: "b5ee1cd9a5e44f139e55591e25a61035" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35165 } } peers { permanent_uuid: "1f5cb71a9b6d4a6cbd272116e69cc67f" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35365 } }
I20260812 01:01:42.314082 13590 raft_consensus.cc:374] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:01:42.314193 13590 raft_consensus.cc:729] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 471561b216ca438bb4464871c8a516b3, State: Initialized, Role: FOLLOWER
I20260812 01:01:42.314877 13590 consensus_queue.cc:260] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471561b216ca438bb4464871c8a516b3" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43425 } } peers { permanent_uuid: "b5ee1cd9a5e44f139e55591e25a61035" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35165 } } peers { permanent_uuid: "1f5cb71a9b6d4a6cbd272116e69cc67f" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35365 } }
I20260812 01:01:42.317479 13591 tablet_bootstrap.cc:492] T 61820fdd49584758ab16f32e7a80cfed P b5ee1cd9a5e44f139e55591e25a61035: No bootstrap required, opened a new log
I20260812 01:01:42.317817 13591 ts_tablet_manager.cc:1397] T 61820fdd49584758ab16f32e7a80cfed P b5ee1cd9a5e44f139e55591e25a61035: Time spent bootstrapping tablet: real 0.036s user 0.011s sys 0.016s
I20260812 01:01:42.328553 13069 heartbeater.cc:503] Master 127.21.201.126:38675 was elected leader, sending a full tablet report...
I20260812 01:01:42.328780 13588 ts_tablet_manager.cc:1428] T 61820fdd49584758ab16f32e7a80cfed P 1f5cb71a9b6d4a6cbd272116e69cc67f: Time spent starting tablet: real 0.039s user 0.007s sys 0.007s
I20260812 01:01:42.328613 13591 raft_consensus.cc:348] T 61820fdd49584758ab16f32e7a80cfed P b5ee1cd9a5e44f139e55591e25a61035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471561b216ca438bb4464871c8a516b3" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43425 } } peers { permanent_uuid: "b5ee1cd9a5e44f139e55591e25a61035" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35165 } } peers { permanent_uuid: "1f5cb71a9b6d4a6cbd272116e69cc67f" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35365 } }
I20260812 01:01:42.329056 13591 raft_consensus.cc:374] T 61820fdd49584758ab16f32e7a80cfed P b5ee1cd9a5e44f139e55591e25a61035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:01:42.329154 13591 raft_consensus.cc:729] T 61820fdd49584758ab16f32e7a80cfed P b5ee1cd9a5e44f139e55591e25a61035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5ee1cd9a5e44f139e55591e25a61035, State: Initialized, Role: FOLLOWER
I20260812 01:01:42.329730 13591 consensus_queue.cc:260] T 61820fdd49584758ab16f32e7a80cfed P b5ee1cd9a5e44f139e55591e25a61035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471561b216ca438bb4464871c8a516b3" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43425 } } peers { permanent_uuid: "b5ee1cd9a5e44f139e55591e25a61035" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35165 } } peers { permanent_uuid: "1f5cb71a9b6d4a6cbd272116e69cc67f" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35365 } }
I20260812 01:01:42.329998 13590 ts_tablet_manager.cc:1428] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3: Time spent starting tablet: real 0.041s user 0.013s sys 0.001s
I20260812 01:01:42.330951 13313 heartbeater.cc:503] Master 127.21.201.126:38675 was elected leader, sending a full tablet report...
I20260812 01:01:42.332218 13547 heartbeater.cc:503] Master 127.21.201.126:38675 was elected leader, sending a full tablet report...
I20260812 01:01:42.334211 13591 ts_tablet_manager.cc:1428] T 61820fdd49584758ab16f32e7a80cfed P b5ee1cd9a5e44f139e55591e25a61035: Time spent starting tablet: real 0.016s user 0.011s sys 0.006s
I20260812 01:01:42.421334 13604 raft_consensus.cc:482] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:01:42.421694 13604 raft_consensus.cc:504] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471561b216ca438bb4464871c8a516b3" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43425 } } peers { permanent_uuid: "b5ee1cd9a5e44f139e55591e25a61035" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35165 } } peers { permanent_uuid: "1f5cb71a9b6d4a6cbd272116e69cc67f" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35365 } }
I20260812 01:01:42.423553 13604 leader_election.cc:288] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5ee1cd9a5e44f139e55591e25a61035 (127.21.201.67:35165), 1f5cb71a9b6d4a6cbd272116e69cc67f (127.21.201.66:35365)
I20260812 01:01:42.431340 13584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:42.424149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:59103 (local address 127.21.201.67:35165)
0812 01:01:42.424380 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:42.424389 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:42.424893 (+ 504us) server_negotiation.cc:407] Connection header received
0812 01:01:42.425199 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:42.425205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:42.425285 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:42.425362 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:01:42.426357 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.427371 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.429391 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.430149 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.430349 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:42.430432 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:42.430926 (+ 494us) server_negotiation.cc:299] Negotiation successful
0812 01:01:42.431070 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 01:01:42.431423 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:42.423896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:35165 (local address 127.21.201.65:59103)
0812 01:01:42.424765 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:42.424786 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:42.425019 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:42.425629 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:42.425636 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:42.425684 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:01:42.426155 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:42.426169 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.427491 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.427499 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:42.429228 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:42.429240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.430401 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.430408 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:42.430647 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:42.430674 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 01:01:42.430868 (+ 194us) client_negotiation.cc:240] Negotiation successful
0812 01:01:42.431007 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":50,"thread_start_us":645,"threads_started":1}
I20260812 01:01:42.432678 13471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61820fdd49584758ab16f32e7a80cfed" candidate_uuid: "471561b216ca438bb4464871c8a516b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5ee1cd9a5e44f139e55591e25a61035" is_pre_election: true
I20260812 01:01:42.433264 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:42.425439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:37203 (local address 127.21.201.66:35365)
0812 01:01:42.425644 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:42.425651 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:42.426077 (+ 426us) server_negotiation.cc:407] Connection header received
0812 01:01:42.426350 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:42.426356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:42.426446 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:42.426549 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:01:42.427168 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.428282 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.430807 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.432338 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.432526 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:42.432598 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:42.432878 (+ 280us) server_negotiation.cc:299] Negotiation successful
0812 01:01:42.433007 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 01:01:42.433262 13471 raft_consensus.cc:2454] T 61820fdd49584758ab16f32e7a80cfed P b5ee1cd9a5e44f139e55591e25a61035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 471561b216ca438bb4464871c8a516b3 in term 0.
I20260812 01:01:42.433391 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:42.424982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:35365 (local address 127.21.201.65:37203)
0812 01:01:42.425958 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:42.425983 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:42.426182 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:42.426584 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:42.426589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:42.426618 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:01:42.426988 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:42.426999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.428475 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.428481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:42.430638 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:42.430653 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.432547 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.432554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:42.432715 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:42.432737 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 01:01:42.432854 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 01:01:42.432983 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":27,"thread_start_us":945,"threads_started":1}
I20260812 01:01:42.434000 12884 leader_election.cc:302] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 471561b216ca438bb4464871c8a516b3, b5ee1cd9a5e44f139e55591e25a61035; no voters:
I20260812 01:01:42.434458 13604 raft_consensus.cc:2781] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:01:42.434620 13604 raft_consensus.cc:482] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:01:42.434399 13249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61820fdd49584758ab16f32e7a80cfed" candidate_uuid: "471561b216ca438bb4464871c8a516b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5cb71a9b6d4a6cbd272116e69cc67f" is_pre_election: true
I20260812 01:01:42.434726 13604 raft_consensus.cc:3037] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:01:42.434993 13249 raft_consensus.cc:2454] T 61820fdd49584758ab16f32e7a80cfed P 1f5cb71a9b6d4a6cbd272116e69cc67f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 471561b216ca438bb4464871c8a516b3 in term 0.
I20260812 01:01:42.437392 13604 raft_consensus.cc:504] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471561b216ca438bb4464871c8a516b3" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43425 } } peers { permanent_uuid: "b5ee1cd9a5e44f139e55591e25a61035" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35165 } } peers { permanent_uuid: "1f5cb71a9b6d4a6cbd272116e69cc67f" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35365 } }
I20260812 01:01:42.438071 13604 leader_election.cc:288] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3 [CANDIDATE]: Term 1 election: Requested vote from peers b5ee1cd9a5e44f139e55591e25a61035 (127.21.201.67:35165), 1f5cb71a9b6d4a6cbd272116e69cc67f (127.21.201.66:35365)
I20260812 01:01:42.438481 13471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61820fdd49584758ab16f32e7a80cfed" candidate_uuid: "471561b216ca438bb4464871c8a516b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5ee1cd9a5e44f139e55591e25a61035"
I20260812 01:01:42.438643 13249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61820fdd49584758ab16f32e7a80cfed" candidate_uuid: "471561b216ca438bb4464871c8a516b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5cb71a9b6d4a6cbd272116e69cc67f"
I20260812 01:01:42.438711 13471 raft_consensus.cc:3037] T 61820fdd49584758ab16f32e7a80cfed P b5ee1cd9a5e44f139e55591e25a61035 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:01:42.438854 13249 raft_consensus.cc:3037] T 61820fdd49584758ab16f32e7a80cfed P 1f5cb71a9b6d4a6cbd272116e69cc67f [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:01:42.441071 13471 raft_consensus.cc:2454] T 61820fdd49584758ab16f32e7a80cfed P b5ee1cd9a5e44f139e55591e25a61035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 471561b216ca438bb4464871c8a516b3 in term 1.
I20260812 01:01:42.441497 12884 leader_election.cc:302] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 471561b216ca438bb4464871c8a516b3, b5ee1cd9a5e44f139e55591e25a61035; no voters:
I20260812 01:01:42.441795 13604 raft_consensus.cc:2781] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:01:42.442131 13249 raft_consensus.cc:2454] T 61820fdd49584758ab16f32e7a80cfed P 1f5cb71a9b6d4a6cbd272116e69cc67f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 471561b216ca438bb4464871c8a516b3 in term 1.
I20260812 01:01:42.442353 13604 raft_consensus.cc:686] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3 [term 1 LEADER]: Becoming Leader. State: Replica: 471561b216ca438bb4464871c8a516b3, State: Running, Role: LEADER
I20260812 01:01:42.443207 13604 consensus_queue.cc:237] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471561b216ca438bb4464871c8a516b3" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43425 } } peers { permanent_uuid: "b5ee1cd9a5e44f139e55591e25a61035" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35165 } } peers { permanent_uuid: "1f5cb71a9b6d4a6cbd272116e69cc67f" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35365 } }
I20260812 01:01:42.448324 12731 catalog_manager.cc:5495] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 471561b216ca438bb4464871c8a516b3 (127.21.201.65). New cstate: current_term: 1 leader_uuid: "471561b216ca438bb4464871c8a516b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471561b216ca438bb4464871c8a516b3" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5ee1cd9a5e44f139e55591e25a61035" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f5cb71a9b6d4a6cbd272116e69cc67f" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35365 } health_report { overall_health: UNKNOWN } } }
I20260812 01:01:42.497022 13580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:42.491408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35360 (local address 127.21.201.65:43425)
0812 01:01:42.491643 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:42.491650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:42.491770 (+ 120us) server_negotiation.cc:407] Connection header received
0812 01:01:42.492051 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:42.492055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:42.492159 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:42.492318 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:01:42.492714 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.493457 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.494840 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.495243 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.495395 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:42.495920 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:01:42.496420 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:01:42.496579 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:42.496663 (+ 84us) server_negotiation.cc:299] Negotiation successful
0812 01:01:42.496744 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 01:01:42.505287 13249 raft_consensus.cc:1261] T 61820fdd49584758ab16f32e7a80cfed P 1f5cb71a9b6d4a6cbd272116e69cc67f [term 1 FOLLOWER]: Refusing update from remote peer 471561b216ca438bb4464871c8a516b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:01:42.505574 13471 raft_consensus.cc:1261] T 61820fdd49584758ab16f32e7a80cfed P b5ee1cd9a5e44f139e55591e25a61035 [term 1 FOLLOWER]: Refusing update from remote peer 471561b216ca438bb4464871c8a516b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:01:42.506389 13627 consensus_queue.cc:1035] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f5cb71a9b6d4a6cbd272116e69cc67f" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:01:42.506980 13604 consensus_queue.cc:1035] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5ee1cd9a5e44f139e55591e25a61035" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:01:42.521727 13631 mvcc.cc:205] Tried to move back new op lower bound from 7317489674251173888 to 7317489674011512832. Current Snapshot: MvccSnapshot[applied={T|T < 7317489674251173888}]
I20260812 01:01:42.523727 13634 mvcc.cc:205] Tried to move back new op lower bound from 7317489674251173888 to 7317489674011512832. Current Snapshot: MvccSnapshot[applied={T|T < 7317489674251173888}]
I20260812 01:01:42.540639 12731 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:01:42.540938 12731 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:01:42.544515 12731 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e0928e827fe43ae9ae7af7ca7fbb828: Sending DeleteTablet for 3 replicas of tablet 61820fdd49584758ab16f32e7a80cfed
I20260812 01:01:42.545711 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 61820fdd49584758ab16f32e7a80cfed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:01:42 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:35348
I20260812 01:01:42.546416 13440 tablet_service.cc:1514] Processing DeleteTablet for tablet 61820fdd49584758ab16f32e7a80cfed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:01:42 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:53932
I20260812 01:01:42.547240 13209 tablet_service.cc:1514] Processing DeleteTablet for tablet 61820fdd49584758ab16f32e7a80cfed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:01:42 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:48418
I20260812 01:01:42.560791 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 12792
I20260812 01:01:42.562775 13649 tablet_replica.cc:330] T 61820fdd49584758ab16f32e7a80cfed P b5ee1cd9a5e44f139e55591e25a61035: stopping tablet replica
I20260812 01:01:42.563966 13650 tablet_replica.cc:330] T 61820fdd49584758ab16f32e7a80cfed P 1f5cb71a9b6d4a6cbd272116e69cc67f: stopping tablet replica
I20260812 01:01:42.565438 13650 raft_consensus.cc:2229] T 61820fdd49584758ab16f32e7a80cfed P 1f5cb71a9b6d4a6cbd272116e69cc67f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:01:42.566030 13649 raft_consensus.cc:2229] T 61820fdd49584758ab16f32e7a80cfed P b5ee1cd9a5e44f139e55591e25a61035 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:01:42.567096 13650 raft_consensus.cc:2258] T 61820fdd49584758ab16f32e7a80cfed P 1f5cb71a9b6d4a6cbd272116e69cc67f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:01:42.568217 13649 raft_consensus.cc:2258] T 61820fdd49584758ab16f32e7a80cfed P b5ee1cd9a5e44f139e55591e25a61035 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:01:42.572970 13649 ts_tablet_manager.cc:1905] T 61820fdd49584758ab16f32e7a80cfed P b5ee1cd9a5e44f139e55591e25a61035: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:01:42.582974 13649 ts_tablet_manager.cc:1918] T 61820fdd49584758ab16f32e7a80cfed P b5ee1cd9a5e44f139e55591e25a61035: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:01:42.588091 13649 log.cc:1196] T 61820fdd49584758ab16f32e7a80cfed P b5ee1cd9a5e44f139e55591e25a61035: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/61820fdd49584758ab16f32e7a80cfed
I20260812 01:01:42.589126 13649 ts_tablet_manager.cc:1939] T 61820fdd49584758ab16f32e7a80cfed P b5ee1cd9a5e44f139e55591e25a61035: Deleting consensus metadata
I20260812 01:01:42.591943 12703 catalog_manager.cc:4841] TS b5ee1cd9a5e44f139e55591e25a61035 (127.21.201.67:35165): tablet 61820fdd49584758ab16f32e7a80cfed (table test-table [id=b3e8f19d360a4eae9cbaf332099179ef]) successfully deleted
I20260812 01:01:42.592475 13650 ts_tablet_manager.cc:1905] T 61820fdd49584758ab16f32e7a80cfed P 1f5cb71a9b6d4a6cbd272116e69cc67f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:01:42.603960 13647 tablet_replica.cc:330] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3: stopping tablet replica
I20260812 01:01:42.604578 13647 raft_consensus.cc:2229] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3 [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:01:42.605376 13647 raft_consensus.cc:2258] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:01:42.607827 13647 ts_tablet_manager.cc:1905] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:01:42.615394 13650 ts_tablet_manager.cc:1918] T 61820fdd49584758ab16f32e7a80cfed P 1f5cb71a9b6d4a6cbd272116e69cc67f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:01:42.615585 13650 log.cc:1196] T 61820fdd49584758ab16f32e7a80cfed P 1f5cb71a9b6d4a6cbd272116e69cc67f: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/61820fdd49584758ab16f32e7a80cfed
I20260812 01:01:42.616237 13650 ts_tablet_manager.cc:1939] T 61820fdd49584758ab16f32e7a80cfed P 1f5cb71a9b6d4a6cbd272116e69cc67f: Deleting consensus metadata
I20260812 01:01:42.618281 12703 catalog_manager.cc:4841] TS 1f5cb71a9b6d4a6cbd272116e69cc67f (127.21.201.66:35365): tablet 61820fdd49584758ab16f32e7a80cfed (table test-table [id=b3e8f19d360a4eae9cbaf332099179ef]) successfully deleted
I20260812 01:01:42.626603 13647 ts_tablet_manager.cc:1918] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:01:42.629730 13580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:42.564822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35376 (local address 127.21.201.65:43425)
0812 01:01:42.565016 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:42.565027 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:42.589085 (+ 24058us) server_negotiation.cc:407] Connection header received
0812 01:01:42.589711 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:42.589720 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:42.589832 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:42.589924 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:42.598968 (+ 9044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.600760 (+ 1792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.610013 (+ 9253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.610761 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.610910 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:42.619241 (+ 8331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:42.619254 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:42.619271 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:42.619325 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:42.623475 (+ 4150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:42.626870 (+ 3395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:42.626879 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:42.626883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:42.627071 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:42.627542 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:42.627547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:42.627551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:42.628031 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:42.628384 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:42.629016 (+ 632us) server_negotiation.cc:299] Negotiation successful
0812 01:01:42.629258 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 01:01:42.630267 13647 log.cc:1196] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786496278741276-22309-0/minicluster-data/ts-0/wal/wals/61820fdd49584758ab16f32e7a80cfed
I20260812 01:01:42.630743 13061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:01:42.751935 13647 ts_tablet_manager.cc:1939] T 61820fdd49584758ab16f32e7a80cfed P 471561b216ca438bb4464871c8a516b3: Deleting consensus metadata
I20260812 01:01:42.752635 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 12792
I20260812 01:01:42.779990 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 13075
W20260812 01:01:42.779942 12706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.201.65:43425 (error 108)
I20260812 01:01:42.782032 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:42.781262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:43425 (local address 127.0.0.1:35384)
0812 01:01:42.781738 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:42.781852 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:43425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":246,"threads_started":1}
W20260812 01:01:42.782300 12706 catalog_manager.cc:4584] TS 471561b216ca438bb4464871c8a516b3 (127.21.201.65:43425): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 61820fdd49584758ab16f32e7a80cfed: Network error: Client connection negotiation failed: client connection to 127.21.201.65:43425: connect: Connection refused (error 111)
I20260812 01:01:42.794446 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:42.780825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48434 (local address 127.21.201.66:35365)
0812 01:01:42.781085 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:42.781092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:42.781153 (+ 61us) server_negotiation.cc:407] Connection header received
0812 01:01:42.781416 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:42.781422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:42.781506 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:42.781620 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:42.783175 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.784035 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.785395 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.785779 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.785920 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:42.788596 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:42.788609 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:42.788623 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:42.788657 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:42.791407 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:42.792191 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:42.792198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:42.792201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:42.792334 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:42.792702 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:42.792708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:42.792711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:42.793190 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:42.793472 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:42.793799 (+ 327us) server_negotiation.cc:299] Negotiation successful
0812 01:01:42.794018 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260812 01:01:42.795156 13307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:01:42.835708 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:42.835276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:43425 (local address 127.0.0.1:35400)
0812 01:01:42.835479 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:42.835572 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:43425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260812 01:01:42.836859 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:42.836481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:43425 (local address 127.0.0.1:35406)
0812 01:01:42.836664 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:42.836735 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:43425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260812 01:01:42.896348 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 13075
I20260812 01:01:42.900352 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:42.899891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:43425 (local address 127.0.0.1:35412)
0812 01:01:42.900092 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:42.900194 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:43425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260812 01:01:42.901434 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:42.901047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:43425 (local address 127.0.0.1:35422)
0812 01:01:42.901206 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:42.901291 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:43425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 01:01:42.921793 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 13318
I20260812 01:01:42.934938 13584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:42.922603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53934 (local address 127.21.201.67:35165)
0812 01:01:42.922776 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:42.922781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:42.922800 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:01:42.922951 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:42.922955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:42.923037 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:42.923123 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:42.924557 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.925277 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.926479 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:42.926864 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:42.926957 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:42.929616 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:42.929631 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:42.929648 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:42.929679 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:42.932633 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:42.933220 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:42.933226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:42.933229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:42.933362 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:42.933673 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:42.933676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:42.933678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:42.934030 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:42.934241 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:42.934508 (+ 267us) server_negotiation.cc:299] Negotiation successful
0812 01:01:42.934660 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 01:01:42.935705 13538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:01:42.981803 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:42.981249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:43425 (local address 127.0.0.1:35426)
0812 01:01:42.981505 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:42.981618 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:43425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260812 01:01:42.983035 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:42.982528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:43425 (local address 127.0.0.1:35430)
0812 01:01:42.982757 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:42.982850 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:43425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260812 01:01:43.025503 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 13318
I20260812 01:01:43.048651 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 12670
I20260812 01:01:43.049876 12783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:01:43.166101 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:43.165514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:43425 (local address 127.0.0.1:35432)
0812 01:01:43.165762 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:43.165909 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:43425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260812 01:01:43.166103 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 12670
I20260812 01:01:43.199558 22309 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 01:01:43.544763 22309 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 01:01:43.545024 22309 mini_ranger.cc:66] Stopping Ranger...
I20260812 01:01:43.901149 22309 mini_ranger.cc:68] Stopped Ranger
2026-08-12 01:01:43.901 UTC [10905] LOG: received smart shutdown request
2026-08-12 01:01:43.904 UTC [10905] LOG: background worker "logical replication launcher" (PID 10917) exited with exit code 1
2026-08-12 01:01:43.904 UTC [10909] LOG: shutting down
2026-08-12 01:01:44.085 UTC [10905] LOG: database system is shut down
2026-08-12T01:01:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55880 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:01:44 dist-test-slave-7f01 krb5kdc[13675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:01:44 dist-test-slave-7f01 krb5kdc[13675](info): set up 2 sockets
Aug 12 01:01:44 dist-test-slave-7f01 krb5kdc[13675](info): commencing operation
krb5kdc: starting...
W20260812 01:01:45.572676 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.307s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:01:45 dist-test-slave-7f01 krb5kdc[13675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:01:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:01:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 01:01:45.855351 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:41039
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:42785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:41039,127.21.201.125:45007,127.21.201.124:44523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:01:46.203913 13697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:01:46.204221 13697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:01:46.204402 13697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:01:46.211882 13697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:01:46.211997 13697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:01:46.212097 13697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:01:46.212210 13697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:01:46.212291 13697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:01:46.222625 13697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:42785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:41039,127.21.201.125:45007,127.21.201.124:44523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:41039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:01:46.224049 13697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:01:46.225530 13697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:01:46.238036 13704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:46.237985 13709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:46.238497 13705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:01:46.239317 13697 server_base.cc:1034] running on GCE node
I20260812 01:01:46.240217 13697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:01:46.241581 13697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:01:46.242837 13697 hybrid_clock.cc:648] HybridClock initialized: now 1786496506242806 us; error 49 us; skew 500 ppm
I20260812 01:01:46.243264 13697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:01:46 dist-test-slave-7f01 krb5kdc[13675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496506, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:01:46.252230 13697 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 01:01:46.253713 13697 webserver.cc:460] Webserver started at http://127.21.201.126:46801/ using document root <none> and password file <none>
I20260812 01:01:46.254446 13697 fs_manager.cc:359] Metadata directory not provided
I20260812 01:01:46.254603 13697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:01:46.254927 13697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:01:46.258028 13697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "caf3a11394944625876aeb85be168472"
format_stamp: "Formatted at 2026-08-12 01:01:46 on dist-test-slave-7f01"
server_key: "b5b85f6e56cb4e32672345a3fdf64554"
server_key_iv: "8d9d00afc763040ccdf37da4a02fb612"
server_key_version: "encryptionkey@0"
I20260812 01:01:46.258852 13697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "caf3a11394944625876aeb85be168472"
format_stamp: "Formatted at 2026-08-12 01:01:46 on dist-test-slave-7f01"
server_key: "b5b85f6e56cb4e32672345a3fdf64554"
server_key_iv: "8d9d00afc763040ccdf37da4a02fb612"
server_key_version: "encryptionkey@0"
I20260812 01:01:46.264249 13697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 01:01:46.267670 13723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:46.269024 13697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:01:46.269215 13697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "caf3a11394944625876aeb85be168472"
format_stamp: "Formatted at 2026-08-12 01:01:46 on dist-test-slave-7f01"
server_key: "b5b85f6e56cb4e32672345a3fdf64554"
server_key_iv: "8d9d00afc763040ccdf37da4a02fb612"
server_key_version: "encryptionkey@0"
I20260812 01:01:46.269423 13697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:01:46.282663 13697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:01:46.287087 13697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:01:46.287391 13697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:01:46.307153 13697 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:41039
I20260812 01:01:46.307196 13813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:41039 every 8 connection(s)
I20260812 01:01:46.308585 13697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:01:46.312211 13814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:01:46.312281 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 13697
I20260812 01:01:46.312546 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:01:46.313222 22309 external_mini_cluster.cc:1428] Setting key 9f9275447ce164184d096f89d7dc6f7e
I20260812 01:01:46.319092 13814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41039 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:01:46 dist-test-slave-7f01 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496506, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.201.125@KRBTEST.COM; defaulting to no policy
I20260812 01:01:46.344319 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:46.322187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46910 (local address 127.21.201.126:41039)
0812 01:01:46.324355 (+ 2168us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:46.324365 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:46.324427 (+ 62us) server_negotiation.cc:407] Connection header received
0812 01:01:46.325795 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:46.325806 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:46.326102 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:46.326309 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:46.327476 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:46.329478 (+ 2002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:46.330863 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:46.331910 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:46.332132 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:46.335851 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:46.335865 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:46.335884 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:46.335936 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:46.340159 (+ 4223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:46.340854 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:46.340865 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:46.340871 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:46.341092 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:46.341491 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:46.341497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:46.341500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:46.342169 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:46.342323 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:46.342969 (+ 646us) server_negotiation.cc:299] Negotiation successful
0812 01:01:46.343210 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1929,"spinlock_wait_cycles":13568,"thread_start_us":1693,"threads_started":1}
I20260812 01:01:46.344374 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:46.322188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:41039 (local address 127.0.0.1:46910)
0812 01:01:46.323216 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:46.323275 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:46.325429 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:46.326347 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:46.326351 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:46.326929 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:01:46.327262 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:46.327278 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:46.329643 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:46.329647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:46.330709 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:46.330722 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:46.332169 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:46.332174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:46.332312 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:46.333013 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:01:46.333035 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:01:46.335573 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:01:46.340294 (+ 4721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:46.340304 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:46.340313 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:46.340708 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:46.341225 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:46.341228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:46.341231 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:46.341381 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:46.342341 (+ 960us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:01:46.342348 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:01:46.342531 (+ 183us) client_negotiation.cc:769] Sending connection context
0812 01:01:46.342852 (+ 321us) client_negotiation.cc:240] Negotiation successful
0812 01:01:46.343374 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"mutex_wait_us":107,"thread_start_us":426,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.125@KRBTEST.COM" created.
I20260812 01:01:46.347409 13814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } has no permanent_uuid. Determining permanent_uuid...
I20260812 01:01:46.348672 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:46.348267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:45007 (local address 127.0.0.1:43900)
0812 01:01:46.348452 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:46.348567 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260812 01:01:46.349072 13732 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.201.125:45007: connect: Connection refused (error 111)
I20260812 01:01:46.350626 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:46.350294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:45007 (local address 127.0.0.1:43906)
0812 01:01:46.350433 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:46.350500 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 01:01:46.350931 13814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.125:45007: Network error: Client connection negotiation failed: client connection to 127.21.201.125:45007: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.125.keytab.
Entry for principal kudu/127.21.201.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.125.keytab.
I20260812 01:01:46.377283 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.125:45007
--webserver_interface=127.21.201.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.125.keytab
--principal=kudu/127.21.201.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:42785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:41039,127.21.201.125:45007,127.21.201.124:44523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:01:46.400327 13814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } attempt: 1
I20260812 01:01:46.401613 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:46.401140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:45007 (local address 127.0.0.1:43918)
0812 01:01:46.401355 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:46.401478 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260812 01:01:46.402688 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:46.402365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:45007 (local address 127.0.0.1:43934)
0812 01:01:46.402493 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:46.402562 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 01:01:46.402964 13814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.125:45007: Network error: Client connection negotiation failed: client connection to 127.21.201.125:45007: connect: Connection refused (error 111)
I20260812 01:01:46.471282 13814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } attempt: 2
I20260812 01:01:46.473049 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:46.472296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:45007 (local address 127.0.0.1:43940)
0812 01:01:46.472715 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:46.472851 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260812 01:01:46.474172 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:46.473720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:45007 (local address 127.0.0.1:43954)
0812 01:01:46.473908 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:46.473994 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260812 01:01:46.474386 13814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.125:45007: Network error: Client connection negotiation failed: client connection to 127.21.201.125:45007: connect: Connection refused (error 111)
I20260812 01:01:46.565653 13814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } attempt: 3
I20260812 01:01:46.566830 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:46.566341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:45007 (local address 127.0.0.1:43962)
0812 01:01:46.566564 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:46.566671 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260812 01:01:46.567983 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:46.567565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:45007 (local address 127.0.0.1:43978)
0812 01:01:46.567723 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:46.567852 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260812 01:01:46.568173 13814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.125:45007: Network error: Client connection negotiation failed: client connection to 127.21.201.125:45007: connect: Connection refused (error 111)
I20260812 01:01:46.711400 13814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } attempt: 4
I20260812 01:01:46.712702 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:46.712148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:45007 (local address 127.0.0.1:43994)
0812 01:01:46.712465 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:46.712560 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260812 01:01:46.713861 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:46.713436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:45007 (local address 127.0.0.1:43998)
0812 01:01:46.713615 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:46.713709 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260812 01:01:46.714066 13814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.125:45007: Network error: Client connection negotiation failed: client connection to 127.21.201.125:45007: connect: Connection refused (error 111)
W20260812 01:01:46.714910 13827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:01:46.715211 13827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:01:46.715408 13827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:01:46.722587 13827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:01:46.722730 13827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:01:46.722833 13827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:01:46.722934 13827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:01:46.723016 13827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:01:46.732697 13827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:42785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:41039,127.21.201.125:45007,127.21.201.124:44523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.125.keytab
--principal=kudu/127.21.201.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.125:45007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.201.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:01:46.733973 13827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:01:46.735244 13827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:01:46.747057 13835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:46.747083 13841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:46.747108 13839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:01:46.748217 13827 server_base.cc:1034] running on GCE node
I20260812 01:01:46.748968 13827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:01:46.750227 13827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:01:46.751436 13827 hybrid_clock.cc:648] HybridClock initialized: now 1786496506751397 us; error 32 us; skew 500 ppm
I20260812 01:01:46.751717 13827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:01:46 dist-test-slave-7f01 krb5kdc[13675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496506, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:01:46.754823 13827 init.cc:377] Logged in from keytab as kudu/127.21.201.125@KRBTEST.COM (short username kudu)
I20260812 01:01:46.756203 13827 webserver.cc:460] Webserver started at http://127.21.201.125:46065/ using document root <none> and password file <none>
I20260812 01:01:46.756799 13827 fs_manager.cc:359] Metadata directory not provided
I20260812 01:01:46.756899 13827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:01:46.757139 13827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:01:46.759996 13827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-1/data/instance:
uuid: "afceb7844a9b45a791ea3fce293a43d8"
format_stamp: "Formatted at 2026-08-12 01:01:46 on dist-test-slave-7f01"
server_key: "5076bf682f495c5e62f1ed350045e209"
server_key_iv: "6aec737453f6befab68ac697f701eced"
server_key_version: "encryptionkey@0"
I20260812 01:01:46.760799 13827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-1/wal/instance:
uuid: "afceb7844a9b45a791ea3fce293a43d8"
format_stamp: "Formatted at 2026-08-12 01:01:46 on dist-test-slave-7f01"
server_key: "5076bf682f495c5e62f1ed350045e209"
server_key_iv: "6aec737453f6befab68ac697f701eced"
server_key_version: "encryptionkey@0"
I20260812 01:01:46.765975 13827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:01:46.769430 13855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:46.770589 13827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 01:01:46.770756 13827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-1/wal
uuid: "afceb7844a9b45a791ea3fce293a43d8"
format_stamp: "Formatted at 2026-08-12 01:01:46 on dist-test-slave-7f01"
server_key: "5076bf682f495c5e62f1ed350045e209"
server_key_iv: "6aec737453f6befab68ac697f701eced"
server_key_version: "encryptionkey@0"
I20260812 01:01:46.770944 13827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:01:46.794518 13827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:01:46.798545 13827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:01:46.798836 13827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:01:46.817780 13827 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.125:45007
I20260812 01:01:46.817821 13933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.125:45007 every 8 connection(s)
I20260812 01:01:46.819212 13827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-1/data/info.pb
I20260812 01:01:46.821451 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 13827
I20260812 01:01:46.821650 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-1/wal/instance
I20260812 01:01:46.822319 22309 external_mini_cluster.cc:1428] Setting key 7a5c95420563767448dbc71f2a6fc823
I20260812 01:01:46.823227 13934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:01:46.829632 13934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41039 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:01:46 dist-test-slave-7f01 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496506, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.125@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.201.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.124@KRBTEST.COM" created.
I20260812 01:01:46.854234 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:46.832184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46918 (local address 127.21.201.126:41039)
0812 01:01:46.832422 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:46.832429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:46.833785 (+ 1356us) server_negotiation.cc:407] Connection header received
0812 01:01:46.836749 (+ 2964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:46.836754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:46.836828 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:46.836951 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:46.838975 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:46.839904 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:46.842324 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:46.842726 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:46.842819 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:46.847040 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:46.847050 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:46.847057 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:46.847108 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:46.850905 (+ 3797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:46.851674 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:46.851679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:46.851681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:46.851866 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:46.852311 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:46.852314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:46.852316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:46.852591 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:46.852697 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:46.853527 (+ 830us) server_negotiation.cc:299] Negotiation successful
0812 01:01:46.853747 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 01:01:46.855294 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:46.832740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:41039 (local address 127.0.0.1:46918)
0812 01:01:46.833609 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:46.833657 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:46.836424 (+ 2767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:46.837171 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:46.837184 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:46.838009 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:01:46.838757 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:46.838784 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:46.840070 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:46.840078 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:46.842174 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:46.842192 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:46.842971 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:46.842980 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:46.843283 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:46.844076 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:01:46.844119 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:01:46.846797 (+ 2678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:01:46.851081 (+ 4284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:46.851092 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:46.851108 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:46.851560 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:46.851999 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:46.852006 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:46.852013 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:46.852211 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:46.852705 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:01:46.852712 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:01:46.853176 (+ 464us) client_negotiation.cc:769] Sending connection context
0812 01:01:46.853419 (+ 243us) client_negotiation.cc:240] Negotiation successful
0812 01:01:46.853939 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":345,"threads_started":1}
I20260812 01:01:46.857146 13934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:01:46 dist-test-slave-7f01 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496506, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.125@KRBTEST.COM for kudu/127.21.201.125@KRBTEST.COM
I20260812 01:01:46.873202 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:46.858122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:45007 (local address 127.0.0.1:44000)
0812 01:01:46.858289 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:46.858310 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:46.858478 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:46.859210 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:46.859214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:46.859662 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:01:46.859853 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:46.859863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:46.861188 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:46.861196 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:46.862038 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:46.862050 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:46.862840 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:46.862846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:46.863190 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:46.863919 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:01:46.863938 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:01:46.865957 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:01:46.870453 (+ 4496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:46.870457 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:46.870460 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:46.870831 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:46.871290 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:46.871293 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:46.871295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:46.871459 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:46.872307 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:01:46.872311 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:01:46.872445 (+ 134us) client_negotiation.cc:769] Sending connection context
0812 01:01:46.872625 (+ 180us) client_negotiation.cc:240] Negotiation successful
0812 01:01:46.872894 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260812 01:01:46.873705 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:46.858207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44000 (local address 127.21.201.125:45007)
0812 01:01:46.858806 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:46.858812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:46.858835 (+ 23us) server_negotiation.cc:407] Connection header received
0812 01:01:46.858898 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:46.858904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:46.859061 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:46.859218 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:46.860019 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:46.861051 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:46.862184 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:46.862696 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:46.862831 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:46.866206 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:46.866222 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:46.866241 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:46.866300 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:46.870271 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:46.870998 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:46.871009 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:46.871015 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:46.871172 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:46.871571 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:46.871579 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:46.871585 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:46.872142 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:46.872374 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:46.872811 (+ 437us) server_negotiation.cc:299] Negotiation successful
0812 01:01:46.873096 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":293,"threads_started":1}
I20260812 01:01:46.876556 13934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } has no permanent_uuid. Determining permanent_uuid...
I20260812 01:01:46.877669 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:46.877241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:44523 (local address 127.0.0.1:36252)
0812 01:01:46.877425 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:46.877525 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260812 01:01:46.878043 13857 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
I20260812 01:01:46.879675 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:46.879304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:44523 (local address 127.0.0.1:36258)
0812 01:01:46.879495 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:46.879572 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260812 01:01:46.880036 13934 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:44523: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.124.keytab.
Entry for principal kudu/127.21.201.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.124.keytab.
I20260812 01:01:46.891644 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.124:44523
--webserver_interface=127.21.201.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.124.keytab
--principal=kudu/127.21.201.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:42785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:41039,127.21.201.125:45007,127.21.201.124:44523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:01:46.929352 13934 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } attempt: 1
I20260812 01:01:46.930678 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:46.930174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:44523 (local address 127.0.0.1:36270)
0812 01:01:46.930398 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:46.930509 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260812 01:01:46.931886 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:46.931426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:44523 (local address 127.0.0.1:36282)
0812 01:01:46.931624 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:46.931702 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260812 01:01:46.932107 13934 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:44523: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
I20260812 01:01:47.000356 13934 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } attempt: 2
I20260812 01:01:47.001612 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.001105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:44523 (local address 127.0.0.1:36292)
0812 01:01:47.001342 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:47.001445 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260812 01:01:47.002728 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.002392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:44523 (local address 127.0.0.1:36304)
0812 01:01:47.002550 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:47.002615 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 01:01:47.002964 13934 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:44523: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
I20260812 01:01:47.013336 13814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } attempt: 5
Aug 12 01:01:47 dist-test-slave-7f01 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496506, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for kudu/127.21.201.125@KRBTEST.COM
I20260812 01:01:47.028869 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.014083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:45007 (local address 127.0.0.1:44002)
0812 01:01:47.014299 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:47.014329 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:47.014443 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:47.014843 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:47.014846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:47.015232 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:01:47.015430 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:47.015439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.017062 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.017066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:47.017962 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:47.017975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.018972 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.018979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:47.019215 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:47.019901 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:01:47.019919 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:01:47.022286 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:01:47.026308 (+ 4022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:47.026312 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:47.026315 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:47.026719 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:47.027202 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:47.027205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:47.027208 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:47.027357 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:47.028103 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:01:47.028107 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:01:47.028261 (+ 154us) client_negotiation.cc:769] Sending connection context
0812 01:01:47.028390 (+ 129us) client_negotiation.cc:240] Negotiation successful
0812 01:01:47.028615 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260812 01:01:47.029084 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.014223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44002 (local address 127.21.201.125:45007)
0812 01:01:47.014419 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:47.014428 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:47.014458 (+ 30us) server_negotiation.cc:407] Connection header received
0812 01:01:47.014551 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:47.014559 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:47.014675 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:47.014808 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:47.015624 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.016896 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.018121 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.018783 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.018933 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:47.022501 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:47.022508 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:47.022512 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:47.022550 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:47.026156 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:47.026877 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:47.026885 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:47.026890 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:47.027079 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:47.027483 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:47.027490 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:47.027495 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:47.027964 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:47.028108 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:47.028528 (+ 420us) server_negotiation.cc:299] Negotiation successful
0812 01:01:47.028761 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 01:01:47.030380 13814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } has no permanent_uuid. Determining permanent_uuid...
I20260812 01:01:47.031493 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.031073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:44523 (local address 127.0.0.1:36314)
0812 01:01:47.031260 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:47.031343 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260812 01:01:47.032620 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.032272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:44523 (local address 127.0.0.1:36322)
0812 01:01:47.032417 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:47.032505 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 01:01:47.032920 13814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:44523: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
I20260812 01:01:47.084188 13814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } attempt: 1
I20260812 01:01:47.085461 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.084951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:44523 (local address 127.0.0.1:36324)
0812 01:01:47.085172 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:47.085281 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260812 01:01:47.086477 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.086129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:44523 (local address 127.0.0.1:36338)
0812 01:01:47.086284 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:47.086369 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 01:01:47.086711 13814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:44523: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
I20260812 01:01:47.094172 13934 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } attempt: 3
I20260812 01:01:47.095208 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.094768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:44523 (local address 127.0.0.1:36348)
0812 01:01:47.094961 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:47.095053 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260812 01:01:47.096391 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.095970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:44523 (local address 127.0.0.1:36364)
0812 01:01:47.096206 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:47.096280 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260812 01:01:47.096585 13934 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:44523: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
I20260812 01:01:47.154964 13814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } attempt: 2
I20260812 01:01:47.156195 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.155669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:44523 (local address 127.0.0.1:36372)
0812 01:01:47.155928 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:47.156019 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260812 01:01:47.157254 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.156926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:44523 (local address 127.0.0.1:36376)
0812 01:01:47.157081 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:47.157153 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 01:01:47.157480 13814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:44523: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
W20260812 01:01:47.232906 13944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:01:47.233247 13944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:01:47.233408 13944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260812 01:01:47.239919 13934 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } attempt: 4
W20260812 01:01:47.240574 13944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:01:47.240705 13944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:01:47.240808 13944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:01:47.240918 13944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:01:47.241004 13944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:01:47.241322 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.240653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:44523 (local address 127.0.0.1:35266)
0812 01:01:47.241039 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:47.241149 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260812 01:01:47.242323 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.241998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:44523 (local address 127.0.0.1:35274)
0812 01:01:47.242149 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:47.242221 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260812 01:01:47.242539 13934 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:44523: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
I20260812 01:01:47.250877 13944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:42785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:41039,127.21.201.125:45007,127.21.201.124:44523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.124.keytab
--principal=kudu/127.21.201.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.124:44523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.201.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:01:47.252213 13944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:01:47.253492 13944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 01:01:47.263674 13814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } attempt: 3
I20260812 01:01:47.264892 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.264353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:44523 (local address 127.0.0.1:35276)
0812 01:01:47.264634 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:47.264734 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260812 01:01:47.266034 13951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:01:47.266418 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.265827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:44523 (local address 127.0.0.1:35292)
0812 01:01:47.266103 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:47.266204 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
W20260812 01:01:47.266469 13952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:47.266819 13814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:44523: Network error: Client connection negotiation failed: client connection to 127.21.201.124:44523: connect: Connection refused (error 111)
W20260812 01:01:47.266940 13960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:01:47.267166 13944 server_base.cc:1034] running on GCE node
I20260812 01:01:47.268046 13944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:01:47.269317 13944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:01:47.270526 13944 hybrid_clock.cc:648] HybridClock initialized: now 1786496507270497 us; error 26 us; skew 500 ppm
I20260812 01:01:47.270865 13944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:01:47 dist-test-slave-7f01 krb5kdc[13675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496507, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:01:47.274153 13944 init.cc:377] Logged in from keytab as kudu/127.21.201.124@KRBTEST.COM (short username kudu)
I20260812 01:01:47.275694 13944 webserver.cc:460] Webserver started at http://127.21.201.124:37411/ using document root <none> and password file <none>
I20260812 01:01:47.276415 13944 fs_manager.cc:359] Metadata directory not provided
I20260812 01:01:47.276561 13944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:01:47.276865 13944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:01:47.279775 13944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-2/data/instance:
uuid: "741f68b24e25486c91cd351ba7c5417c"
format_stamp: "Formatted at 2026-08-12 01:01:47 on dist-test-slave-7f01"
server_key: "85142f869f2a6067d4008713cc4f6f65"
server_key_iv: "056ac31d59805c8937f7d761b256cf5e"
server_key_version: "encryptionkey@0"
I20260812 01:01:47.280608 13944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-2/wal/instance:
uuid: "741f68b24e25486c91cd351ba7c5417c"
format_stamp: "Formatted at 2026-08-12 01:01:47 on dist-test-slave-7f01"
server_key: "85142f869f2a6067d4008713cc4f6f65"
server_key_iv: "056ac31d59805c8937f7d761b256cf5e"
server_key_version: "encryptionkey@0"
I20260812 01:01:47.285825 13944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260812 01:01:47.289197 13971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:47.290717 13944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260812 01:01:47.290922 13944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-2/wal
uuid: "741f68b24e25486c91cd351ba7c5417c"
format_stamp: "Formatted at 2026-08-12 01:01:47 on dist-test-slave-7f01"
server_key: "85142f869f2a6067d4008713cc4f6f65"
server_key_iv: "056ac31d59805c8937f7d761b256cf5e"
server_key_version: "encryptionkey@0"
I20260812 01:01:47.291131 13944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:01:47.314402 13944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:01:47.320013 13944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:01:47.320261 13944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:01:47.339427 13944 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.124:44523
I20260812 01:01:47.339486 14061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.124:44523 every 8 connection(s)
I20260812 01:01:47.340914 13944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-2/data/info.pb
I20260812 01:01:47.344821 14063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:01:47.347079 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 13944
I20260812 01:01:47.347328 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786496278741276-22309-0/minicluster-data/master-2/wal/instance
I20260812 01:01:47.347890 22309 external_mini_cluster.cc:1428] Setting key af3e05acb5004a4dfe2aad39e665454f
I20260812 01:01:47.353193 14063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41039 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:01:47 dist-test-slave-7f01 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.124@KRBTEST.COM
Aug 12 01:01:47 dist-test-slave-7f01 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496507, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.124@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:01:47.373080 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.356087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36224 (local address 127.21.201.126:41039)
0812 01:01:47.356610 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:47.356617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:47.357231 (+ 614us) server_negotiation.cc:407] Connection header received
0812 01:01:47.357428 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:47.357433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:47.357524 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:47.357638 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:47.359012 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.360117 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.361998 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.362411 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.362504 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:47.365929 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:47.365936 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:47.365942 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:47.365984 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:47.369624 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:47.370533 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:47.370541 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:47.370548 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:47.370698 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:47.371181 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:47.371186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:47.371190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:47.371562 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:47.371715 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:47.372396 (+ 681us) server_negotiation.cc:299] Negotiation successful
0812 01:01:47.372640 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":256,"threads_started":1}
I20260812 01:01:47.374122 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.349969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35302 (local address 127.21.201.124:44523)
0812 01:01:47.350821 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:47.350836 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:47.350901 (+ 65us) server_negotiation.cc:407] Connection header received
0812 01:01:47.353272 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:47.353299 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:47.353781 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:47.354211 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:47.355623 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.357439 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.359347 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.360376 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.360509 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:47.365527 (+ 5018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:47.365535 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:47.365547 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:47.365581 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:47.369348 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:47.370413 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:47.370420 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:47.370422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:47.370671 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:47.371181 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:47.371186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:47.371190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:47.371861 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:47.372089 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:47.372521 (+ 432us) server_negotiation.cc:299] Negotiation successful
0812 01:01:47.372965 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":143,"server-negotiator.queue_time_us":610,"thread_start_us":370,"threads_started":1}
I20260812 01:01:47.374162 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.356108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:41039 (local address 127.0.0.1:36224)
0812 01:01:47.357098 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:47.357137 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:47.357287 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:47.357728 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:47.357736 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:47.358524 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:01:47.358860 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:47.358872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.360287 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.360290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:47.361840 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:47.361858 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.362647 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.362654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:47.362866 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:47.363500 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:01:47.363531 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:01:47.365739 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:01:47.369771 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:47.369783 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:47.369794 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:47.370411 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:47.370821 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:47.370829 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:47.370835 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:47.371064 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:47.371697 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:01:47.371702 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:01:47.371958 (+ 256us) client_negotiation.cc:769] Sending connection context
0812 01:01:47.372243 (+ 285us) client_negotiation.cc:240] Negotiation successful
0812 01:01:47.372772 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":555,"threads_started":1}
I20260812 01:01:47.375952 14063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:01:47 dist-test-slave-7f01 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496507, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.124@KRBTEST.COM for kudu/127.21.201.125@KRBTEST.COM
Aug 12 01:01:47 dist-test-slave-7f01 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:01:47.391269 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.376869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44534 (local address 127.21.201.125:45007)
0812 01:01:47.377104 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:47.377108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:47.377124 (+ 16us) server_negotiation.cc:407] Connection header received
0812 01:01:47.377175 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:47.377179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:47.377240 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:47.377355 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:47.378303 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.379427 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.381193 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.381887 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.382018 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:47.385298 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:47.385306 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:47.385312 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:47.385360 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:47.388363 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:47.389346 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:47.389350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:47.389352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:47.389499 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:47.389998 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:47.390002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:47.390004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:47.390269 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:47.390392 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:47.390819 (+ 427us) server_negotiation.cc:299] Negotiation successful
0812 01:01:47.390989 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 01:01:47.391408 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.376639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:45007 (local address 127.0.0.1:44534)
0812 01:01:47.376836 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:47.376855 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:47.376957 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:47.377372 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:47.377377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:47.377930 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:01:47.378132 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:47.378147 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.379596 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.379600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:47.381057 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:47.381069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.382046 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.382050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:47.382208 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:47.382777 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:01:47.382809 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:01:47.385085 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:01:47.388597 (+ 3512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:47.388604 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:47.388611 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:47.389194 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:47.389642 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:47.389667 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:47.389673 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:47.389874 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:47.390397 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:01:47.390402 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:01:47.390578 (+ 176us) client_negotiation.cc:769] Sending connection context
0812 01:01:47.390742 (+ 164us) client_negotiation.cc:240] Negotiation successful
0812 01:01:47.391043 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260812 01:01:47.392501 14063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } has no permanent_uuid. Determining permanent_uuid...
I20260812 01:01:47.394776 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.377634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36234 (local address 127.21.201.126:41039)
0812 01:01:47.377805 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:47.377810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:47.377833 (+ 23us) server_negotiation.cc:407] Connection header received
0812 01:01:47.377988 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:47.377993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:47.378081 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:47.378189 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:47.379713 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.380805 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.382502 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.383160 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.383318 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:47.387939 (+ 4621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:47.387944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:47.387947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:47.387979 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:47.391328 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:47.392261 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:47.392266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:47.392270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:47.392406 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:47.392877 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:47.392882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:47.392885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:47.393269 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:47.393563 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:47.394169 (+ 606us) server_negotiation.cc:299] Negotiation successful
0812 01:01:47.394422 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Aug 12 01:01:47 dist-test-slave-7f01 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496507, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.124@KRBTEST.COM for kudu/127.21.201.124@KRBTEST.COM
Aug 12 01:01:47 dist-test-slave-7f01 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.125@KRBTEST.COM
I20260812 01:01:47.407307 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.393192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:44523 (local address 127.0.0.1:35318)
0812 01:01:47.393391 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:47.393411 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:47.393507 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:47.393916 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:47.393919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:47.394407 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:01:47.394647 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:47.394658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.395865 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.395869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:47.396987 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:47.397003 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.397731 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.397738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:47.397954 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:47.398546 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:01:47.398562 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:01:47.400783 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:01:47.404562 (+ 3779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:47.404566 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:47.404568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:47.404945 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:47.405384 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:47.405389 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:47.405392 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:47.405581 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:47.406103 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:01:47.406109 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:01:47.406275 (+ 166us) client_negotiation.cc:769] Sending connection context
0812 01:01:47.406458 (+ 183us) client_negotiation.cc:240] Negotiation successful
0812 01:01:47.406827 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":3}
I20260812 01:01:47.407406 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.393442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35318 (local address 127.21.201.124:44523)
0812 01:01:47.393624 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:47.393631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:47.393649 (+ 18us) server_negotiation.cc:407] Connection header received
0812 01:01:47.393692 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:47.393696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:47.393767 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:47.393860 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:47.394807 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.395672 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.397132 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.397575 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.397668 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:47.400974 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:47.400978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:47.400981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:47.401009 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:47.404428 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:47.405092 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:47.405099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:47.405102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:47.405249 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:47.405698 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:47.405703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:47.405706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:47.405974 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:47.406141 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:47.406559 (+ 418us) server_negotiation.cc:299] Negotiation successful
0812 01:01:47.406974 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":195,"server-negotiator.queue_time_us":64}
I20260812 01:01:47.412299 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.396533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44536 (local address 127.21.201.125:45007)
0812 01:01:47.396680 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:47.396684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:47.396698 (+ 14us) server_negotiation.cc:407] Connection header received
0812 01:01:47.396844 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:47.396847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:47.396917 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:47.396991 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:47.398459 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.399193 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.400750 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.401266 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.401364 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:47.406387 (+ 5023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:47.406396 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:47.406404 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:47.406451 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:47.409553 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:47.410338 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:47.410341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:47.410344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:47.410480 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:47.410922 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:47.410927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:47.410929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:47.411226 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:47.411460 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:47.411827 (+ 367us) server_negotiation.cc:299] Negotiation successful
0812 01:01:47.412006 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 01:01:47.413201 14063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c: Bootstrap starting.
I20260812 01:01:47.416813 14063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c: Neither blocks nor log segments found. Creating new log.
I20260812 01:01:47.416977 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:01:47.417924 14063 log.cc:824] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c: Log is configured to *not* fsync() on all Append() calls
I20260812 01:01:47.421098 14063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c: No bootstrap required, opened a new log
I20260812 01:01:47.432346 14063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf3a11394944625876aeb85be168472" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41039 } } peers { permanent_uuid: "afceb7844a9b45a791ea3fce293a43d8" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } } peers { permanent_uuid: "741f68b24e25486c91cd351ba7c5417c" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } }
I20260812 01:01:47.432976 14063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:01:47.433175 14063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 741f68b24e25486c91cd351ba7c5417c, State: Initialized, Role: FOLLOWER
I20260812 01:01:47.434451 14063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf3a11394944625876aeb85be168472" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41039 } } peers { permanent_uuid: "afceb7844a9b45a791ea3fce293a43d8" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } } peers { permanent_uuid: "741f68b24e25486c91cd351ba7c5417c" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } }
I20260812 01:01:47.443820 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.422308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36242 (local address 127.21.201.126:41039)
0812 01:01:47.422530 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:47.422536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:47.423148 (+ 612us) server_negotiation.cc:407] Connection header received
0812 01:01:47.423437 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:47.423443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:47.423524 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:47.423598 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:47.426185 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.427333 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.429639 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.430294 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.430413 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:47.434576 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:47.434584 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:47.434588 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:47.434648 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:47.439291 (+ 4643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:47.440679 (+ 1388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:47.440689 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:47.440695 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:47.440878 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:47.441375 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:47.441392 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:47.441398 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:47.441887 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:47.442210 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:47.442914 (+ 704us) server_negotiation.cc:299] Negotiation successful
0812 01:01:47.443230 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 01:01:47.444029 13814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } attempt: 4
I20260812 01:01:47.445819 14085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf3a11394944625876aeb85be168472" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41039 } } peers { permanent_uuid: "afceb7844a9b45a791ea3fce293a43d8" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } } peers { permanent_uuid: "741f68b24e25486c91cd351ba7c5417c" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } } }
I20260812 01:01:47.446523 14085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:01:47.450160 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.422386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44552 (local address 127.21.201.125:45007)
0812 01:01:47.422596 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:47.422605 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:47.429827 (+ 7222us) server_negotiation.cc:407] Connection header received
0812 01:01:47.430344 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:47.430349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:47.430452 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:47.430541 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:47.433573 (+ 3032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.434879 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.437329 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.437991 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.438083 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:47.441645 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:47.441654 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:47.441661 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:47.441704 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:47.446886 (+ 5182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:47.447778 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:47.447784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:47.447787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:47.447932 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:47.448388 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:47.448395 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:47.448398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:47.448796 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:47.449049 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:47.449627 (+ 578us) server_negotiation.cc:299] Negotiation successful
0812 01:01:47.449885 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 01:01:47.452988 14063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:01:47.453912 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.423593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35322 (local address 127.21.201.124:44523)
0812 01:01:47.426856 (+ 3263us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:47.426866 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:47.434826 (+ 7960us) server_negotiation.cc:407] Connection header received
0812 01:01:47.435229 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:47.435236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:47.435318 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:47.435402 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:47.437333 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.438413 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.440759 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.441486 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.441586 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:47.445589 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:47.445597 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:47.445605 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:47.445643 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:47.449750 (+ 4107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:47.451568 (+ 1818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:47.451574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:47.451581 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:47.451770 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:47.452345 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:47.452351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:47.452354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:47.452664 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:47.452938 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:47.453293 (+ 355us) server_negotiation.cc:299] Negotiation successful
0812 01:01:47.453531 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3087}
Aug 12 01:01:47 dist-test-slave-7f01 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496506, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for kudu/127.21.201.124@KRBTEST.COM
W20260812 01:01:47.465790 14108 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 01:01:47.465967 14108 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 01:01:47.470932 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.444623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:44523 (local address 127.0.0.1:35326)
0812 01:01:47.444981 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:47.444998 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:47.445094 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:47.454458 (+ 9364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:47.454465 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:47.454915 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:01:47.455166 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:47.455183 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.456791 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.456799 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:47.458146 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:47.458167 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.459570 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.459576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:47.459778 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:47.460427 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:01:47.460450 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:01:47.464490 (+ 4040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:01:47.468363 (+ 3873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:47.468371 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:47.468375 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:47.468769 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:47.469170 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:47.469175 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:47.469179 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:47.469315 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:47.469888 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:01:47.469893 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:01:47.470041 (+ 148us) client_negotiation.cc:769] Sending connection context
0812 01:01:47.470190 (+ 149us) client_negotiation.cc:240] Negotiation successful
0812 01:01:47.470513 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225}
I20260812 01:01:47.470930 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.445299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35326 (local address 127.21.201.124:44523)
0812 01:01:47.454157 (+ 8858us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:47.454162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:47.454180 (+ 18us) server_negotiation.cc:407] Connection header received
0812 01:01:47.454227 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:47.454231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:47.454302 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:47.454421 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:47.455373 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.456586 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.458361 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.459202 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.459353 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:47.464703 (+ 5350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:47.464710 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:47.464718 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:47.464757 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:47.468213 (+ 3456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:47.468907 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:47.468914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:47.468917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:47.469048 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:47.469427 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:47.469433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:47.469436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:47.469748 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:47.469900 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:47.470296 (+ 396us) server_negotiation.cc:299] Negotiation successful
0812 01:01:47.470538 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8762,"thread_start_us":9863,"threads_started":1}
I20260812 01:01:47.477804 13814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caf3a11394944625876aeb85be168472: Bootstrap starting.
I20260812 01:01:47.462738 13944 master_runner.cc:186] Error getting master registration for 127.21.201.126:41039: OK, instance_id { permanent_uuid: "caf3a11394944625876aeb85be168472" instance_seqno: 1786496506301538 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 01:01:47.478543 13944 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 01:01:47.481909 13814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P caf3a11394944625876aeb85be168472: Neither blocks nor log segments found. Creating new log.
I20260812 01:01:47.483239 13814 log.cc:824] T 00000000000000000000000000000000 P caf3a11394944625876aeb85be168472: Log is configured to *not* fsync() on all Append() calls
I20260812 01:01:47.486377 13814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caf3a11394944625876aeb85be168472: No bootstrap required, opened a new log
I20260812 01:01:47.497641 13814 raft_consensus.cc:348] T 00000000000000000000000000000000 P caf3a11394944625876aeb85be168472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf3a11394944625876aeb85be168472" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41039 } } peers { permanent_uuid: "afceb7844a9b45a791ea3fce293a43d8" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } } peers { permanent_uuid: "741f68b24e25486c91cd351ba7c5417c" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } }
I20260812 01:01:47.498054 13814 raft_consensus.cc:374] T 00000000000000000000000000000000 P caf3a11394944625876aeb85be168472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:01:47.498191 13814 raft_consensus.cc:729] T 00000000000000000000000000000000 P caf3a11394944625876aeb85be168472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caf3a11394944625876aeb85be168472, State: Initialized, Role: FOLLOWER
I20260812 01:01:47.498858 13814 consensus_queue.cc:260] T 00000000000000000000000000000000 P caf3a11394944625876aeb85be168472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf3a11394944625876aeb85be168472" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41039 } } peers { permanent_uuid: "afceb7844a9b45a791ea3fce293a43d8" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } } peers { permanent_uuid: "741f68b24e25486c91cd351ba7c5417c" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } }
I20260812 01:01:47.499862 14112 sys_catalog.cc:455] T 00000000000000000000000000000000 P caf3a11394944625876aeb85be168472 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf3a11394944625876aeb85be168472" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41039 } } peers { permanent_uuid: "afceb7844a9b45a791ea3fce293a43d8" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } } peers { permanent_uuid: "741f68b24e25486c91cd351ba7c5417c" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } } }
I20260812 01:01:47.500204 14112 sys_catalog.cc:458] T 00000000000000000000000000000000 P caf3a11394944625876aeb85be168472 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:01:47.501201 13814 sys_catalog.cc:564] T 00000000000000000000000000000000 P caf3a11394944625876aeb85be168472 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 01:01:47.513161 14125 catalog_manager.cc:1558] T 00000000000000000000000000000000 P caf3a11394944625876aeb85be168472: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 01:01:47.513422 14125 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 01:01:47.514736 13697 master_runner.cc:186] Error getting master registration for 127.21.201.125:45007: OK, instance_id { permanent_uuid: "afceb7844a9b45a791ea3fce293a43d8" instance_seqno: 1786496506812290 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 01:01:47.530997 13697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 01:01:47.540477 13735 tablet.cc:2369] T 00000000000000000000000000000000 P caf3a11394944625876aeb85be168472: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:01:47.541738 13934 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } attempt: 5
Aug 12 01:01:47 dist-test-slave-7f01 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496506, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.125@KRBTEST.COM for kudu/127.21.201.124@KRBTEST.COM
I20260812 01:01:47.565841 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.543320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:44523 (local address 127.0.0.1:35332)
0812 01:01:47.543509 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:47.543531 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:47.543675 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:47.544133 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:47.544139 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:47.544700 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:01:47.544947 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:47.544961 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.546461 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.546467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:47.547963 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:47.547979 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.549811 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.549821 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:47.550096 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:47.550849 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:01:47.550890 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:01:47.558330 (+ 7440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:01:47.562680 (+ 4350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:47.562686 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:47.562690 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:47.563227 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:47.563718 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:47.563723 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:47.563727 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:47.563966 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:47.564753 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:01:47.564759 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:01:47.564944 (+ 185us) client_negotiation.cc:769] Sending connection context
0812 01:01:47.565109 (+ 165us) client_negotiation.cc:240] Negotiation successful
0812 01:01:47.565422 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260812 01:01:47.567090 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:47.543014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35332 (local address 127.21.201.124:44523)
0812 01:01:47.543244 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:47.543250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:47.543628 (+ 378us) server_negotiation.cc:407] Connection header received
0812 01:01:47.543866 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:47.543878 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:47.543974 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:47.544104 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:47.545137 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.546310 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.548150 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:47.549289 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:47.549623 (+ 334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:47.558573 (+ 8950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:47.558580 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:47.558585 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:47.558626 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:47.562509 (+ 3883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:47.563385 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:47.563391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:47.563398 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:47.563571 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:47.564108 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:47.564114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:47.564118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:47.564513 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:47.564681 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:47.566324 (+ 1643us) server_negotiation.cc:299] Negotiation successful
0812 01:01:47.566610 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260812 01:01:47.572288 13978 tablet.cc:2369] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:01:47.573707 13934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afceb7844a9b45a791ea3fce293a43d8: Bootstrap starting.
I20260812 01:01:47.577905 13934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afceb7844a9b45a791ea3fce293a43d8: Neither blocks nor log segments found. Creating new log.
I20260812 01:01:47.579260 13934 log.cc:824] T 00000000000000000000000000000000 P afceb7844a9b45a791ea3fce293a43d8: Log is configured to *not* fsync() on all Append() calls
I20260812 01:01:47.586716 13934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afceb7844a9b45a791ea3fce293a43d8: No bootstrap required, opened a new log
I20260812 01:01:47.599330 13934 raft_consensus.cc:348] T 00000000000000000000000000000000 P afceb7844a9b45a791ea3fce293a43d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf3a11394944625876aeb85be168472" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41039 } } peers { permanent_uuid: "afceb7844a9b45a791ea3fce293a43d8" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } } peers { permanent_uuid: "741f68b24e25486c91cd351ba7c5417c" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } }
I20260812 01:01:47.599954 13934 raft_consensus.cc:374] T 00000000000000000000000000000000 P afceb7844a9b45a791ea3fce293a43d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:01:47.600126 13934 raft_consensus.cc:729] T 00000000000000000000000000000000 P afceb7844a9b45a791ea3fce293a43d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afceb7844a9b45a791ea3fce293a43d8, State: Initialized, Role: FOLLOWER
I20260812 01:01:47.600843 13934 consensus_queue.cc:260] T 00000000000000000000000000000000 P afceb7844a9b45a791ea3fce293a43d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf3a11394944625876aeb85be168472" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41039 } } peers { permanent_uuid: "afceb7844a9b45a791ea3fce293a43d8" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } } peers { permanent_uuid: "741f68b24e25486c91cd351ba7c5417c" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } }
I20260812 01:01:47.602057 14131 sys_catalog.cc:455] T 00000000000000000000000000000000 P afceb7844a9b45a791ea3fce293a43d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf3a11394944625876aeb85be168472" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41039 } } peers { permanent_uuid: "afceb7844a9b45a791ea3fce293a43d8" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } } peers { permanent_uuid: "741f68b24e25486c91cd351ba7c5417c" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } } }
I20260812 01:01:47.602659 14131 sys_catalog.cc:458] T 00000000000000000000000000000000 P afceb7844a9b45a791ea3fce293a43d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:01:47.603320 13934 sys_catalog.cc:564] T 00000000000000000000000000000000 P afceb7844a9b45a791ea3fce293a43d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:01:47.626277 13827 master_runner.cc:263] No leader master found from master afceb7844a9b45a791ea3fce293a43d8
I20260812 01:01:47.626434 13827 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 01:01:47.629787 14152 catalog_manager.cc:1558] T 00000000000000000000000000000000 P afceb7844a9b45a791ea3fce293a43d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 01:01:47.630110 14152 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 01:01:47.747253 14085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:01:47.747622 14085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf3a11394944625876aeb85be168472" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41039 } } peers { permanent_uuid: "afceb7844a9b45a791ea3fce293a43d8" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } } peers { permanent_uuid: "741f68b24e25486c91cd351ba7c5417c" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } }
I20260812 01:01:47.749336 14085 leader_election.cc:288] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers caf3a11394944625876aeb85be168472 (127.21.201.126:41039), afceb7844a9b45a791ea3fce293a43d8 (127.21.201.125:45007)
I20260812 01:01:47.750587 13773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "741f68b24e25486c91cd351ba7c5417c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caf3a11394944625876aeb85be168472" is_pre_election: true
I20260812 01:01:47.751200 13773 raft_consensus.cc:2454] T 00000000000000000000000000000000 P caf3a11394944625876aeb85be168472 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 741f68b24e25486c91cd351ba7c5417c in term 0.
I20260812 01:01:47.751965 13972 leader_election.cc:302] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 741f68b24e25486c91cd351ba7c5417c, caf3a11394944625876aeb85be168472; no voters:
I20260812 01:01:47.753152 13894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "741f68b24e25486c91cd351ba7c5417c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afceb7844a9b45a791ea3fce293a43d8" is_pre_election: true
I20260812 01:01:47.753727 13894 raft_consensus.cc:2454] T 00000000000000000000000000000000 P afceb7844a9b45a791ea3fce293a43d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 741f68b24e25486c91cd351ba7c5417c in term 0.
I20260812 01:01:47.754431 14085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:01:47.754572 14085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:01:47.754657 14085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:01:47.758757 14085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf3a11394944625876aeb85be168472" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41039 } } peers { permanent_uuid: "afceb7844a9b45a791ea3fce293a43d8" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } } peers { permanent_uuid: "741f68b24e25486c91cd351ba7c5417c" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } }
I20260812 01:01:47.759958 13773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "741f68b24e25486c91cd351ba7c5417c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caf3a11394944625876aeb85be168472"
I20260812 01:01:47.760190 13773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P caf3a11394944625876aeb85be168472 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:01:47.767762 13773 raft_consensus.cc:2454] T 00000000000000000000000000000000 P caf3a11394944625876aeb85be168472 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 741f68b24e25486c91cd351ba7c5417c in term 1.
I20260812 01:01:47.768544 13972 leader_election.cc:302] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 741f68b24e25486c91cd351ba7c5417c, caf3a11394944625876aeb85be168472; no voters:
I20260812 01:01:47.774031 13894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "741f68b24e25486c91cd351ba7c5417c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afceb7844a9b45a791ea3fce293a43d8"
I20260812 01:01:47.774326 13894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afceb7844a9b45a791ea3fce293a43d8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:01:47.778487 13894 raft_consensus.cc:2454] T 00000000000000000000000000000000 P afceb7844a9b45a791ea3fce293a43d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 741f68b24e25486c91cd351ba7c5417c in term 1.
I20260812 01:01:47.779299 14085 leader_election.cc:288] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [CANDIDATE]: Term 1 election: Requested vote from peers caf3a11394944625876aeb85be168472 (127.21.201.126:41039), afceb7844a9b45a791ea3fce293a43d8 (127.21.201.125:45007)
I20260812 01:01:47.779810 14085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:01:47.780324 14085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [term 1 LEADER]: Becoming Leader. State: Replica: 741f68b24e25486c91cd351ba7c5417c, State: Running, Role: LEADER
I20260812 01:01:47.781222 14085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf3a11394944625876aeb85be168472" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41039 } } peers { permanent_uuid: "afceb7844a9b45a791ea3fce293a43d8" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } } peers { permanent_uuid: "741f68b24e25486c91cd351ba7c5417c" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } }
W20260812 01:01:47.802279 13866 tablet.cc:2369] T 00000000000000000000000000000000 P afceb7844a9b45a791ea3fce293a43d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:01:47.804224 14154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 741f68b24e25486c91cd351ba7c5417c. Latest consensus state: current_term: 1 leader_uuid: "741f68b24e25486c91cd351ba7c5417c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf3a11394944625876aeb85be168472" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41039 } } peers { permanent_uuid: "afceb7844a9b45a791ea3fce293a43d8" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } } peers { permanent_uuid: "741f68b24e25486c91cd351ba7c5417c" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } } }
I20260812 01:01:47.804957 14154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [sys.catalog]: This master's current role is: LEADER
I20260812 01:01:47.816295 14157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:01:47.818663 14157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:01:47.844712 13894 raft_consensus.cc:1261] T 00000000000000000000000000000000 P afceb7844a9b45a791ea3fce293a43d8 [term 1 FOLLOWER]: Refusing update from remote peer 741f68b24e25486c91cd351ba7c5417c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:01:47.845760 14154 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [LEADER]: Connected to new peer: Peer: permanent_uuid: "afceb7844a9b45a791ea3fce293a43d8" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:01:47.852000 13773 raft_consensus.cc:1261] T 00000000000000000000000000000000 P caf3a11394944625876aeb85be168472 [term 1 FOLLOWER]: Refusing update from remote peer 741f68b24e25486c91cd351ba7c5417c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:01:47.852996 14154 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [LEADER]: Connected to new peer: Peer: permanent_uuid: "caf3a11394944625876aeb85be168472" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:01:47.879146 14112 sys_catalog.cc:455] T 00000000000000000000000000000000 P caf3a11394944625876aeb85be168472 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 741f68b24e25486c91cd351ba7c5417c. Latest consensus state: current_term: 1 leader_uuid: "741f68b24e25486c91cd351ba7c5417c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf3a11394944625876aeb85be168472" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41039 } } peers { permanent_uuid: "afceb7844a9b45a791ea3fce293a43d8" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } } peers { permanent_uuid: "741f68b24e25486c91cd351ba7c5417c" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } } }
I20260812 01:01:47.879680 14112 sys_catalog.cc:458] T 00000000000000000000000000000000 P caf3a11394944625876aeb85be168472 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:01:47.891155 14158 mvcc.cc:205] Tried to move back new op lower bound from 7317489696121167872 to 7317489695885029376. Current Snapshot: MvccSnapshot[applied={T|T < 7317489696121167872}]
I20260812 01:01:47.893458 14165 mvcc.cc:205] Tried to move back new op lower bound from 7317489696121167872 to 7317489695885029376. Current Snapshot: MvccSnapshot[applied={T|T < 7317489696121167872}]
I20260812 01:01:47.903415 14085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "741f68b24e25486c91cd351ba7c5417c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf3a11394944625876aeb85be168472" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41039 } } peers { permanent_uuid: "afceb7844a9b45a791ea3fce293a43d8" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } } peers { permanent_uuid: "741f68b24e25486c91cd351ba7c5417c" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } } }
I20260812 01:01:47.903800 14085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [sys.catalog]: This master's current role is: LEADER
I20260812 01:01:47.904425 14112 sys_catalog.cc:455] T 00000000000000000000000000000000 P caf3a11394944625876aeb85be168472 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "741f68b24e25486c91cd351ba7c5417c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf3a11394944625876aeb85be168472" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41039 } } peers { permanent_uuid: "afceb7844a9b45a791ea3fce293a43d8" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } } peers { permanent_uuid: "741f68b24e25486c91cd351ba7c5417c" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } } }
I20260812 01:01:47.904654 14157 catalog_manager.cc:1347] Generated new cluster ID: 7aafd197861d43d6a59cc7da8fdf24df
I20260812 01:01:47.904695 14112 sys_catalog.cc:458] T 00000000000000000000000000000000 P caf3a11394944625876aeb85be168472 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:01:47.904770 14157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:01:47.908425 14131 sys_catalog.cc:455] T 00000000000000000000000000000000 P afceb7844a9b45a791ea3fce293a43d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 741f68b24e25486c91cd351ba7c5417c. Latest consensus state: current_term: 1 leader_uuid: "741f68b24e25486c91cd351ba7c5417c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf3a11394944625876aeb85be168472" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41039 } } peers { permanent_uuid: "afceb7844a9b45a791ea3fce293a43d8" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } } peers { permanent_uuid: "741f68b24e25486c91cd351ba7c5417c" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } } }
I20260812 01:01:47.908763 14131 sys_catalog.cc:458] T 00000000000000000000000000000000 P afceb7844a9b45a791ea3fce293a43d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:01:47.921979 14154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "741f68b24e25486c91cd351ba7c5417c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf3a11394944625876aeb85be168472" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41039 } } peers { permanent_uuid: "afceb7844a9b45a791ea3fce293a43d8" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } } peers { permanent_uuid: "741f68b24e25486c91cd351ba7c5417c" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } } }
I20260812 01:01:47.922776 14154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c [sys.catalog]: This master's current role is: LEADER
I20260812 01:01:47.943053 14157 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:01:47.944905 14157 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:01:47.948238 14131 sys_catalog.cc:455] T 00000000000000000000000000000000 P afceb7844a9b45a791ea3fce293a43d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "741f68b24e25486c91cd351ba7c5417c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf3a11394944625876aeb85be168472" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41039 } } peers { permanent_uuid: "afceb7844a9b45a791ea3fce293a43d8" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 45007 } } peers { permanent_uuid: "741f68b24e25486c91cd351ba7c5417c" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 44523 } } }
I20260812 01:01:47.948529 14131 sys_catalog.cc:458] T 00000000000000000000000000000000 P afceb7844a9b45a791ea3fce293a43d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:01:47.968941 14157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c: Generated new TSK 0
I20260812 01:01:47.970098 14157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:01:48.515620 14125 catalog_manager.cc:1259] Loaded cluster ID: 7aafd197861d43d6a59cc7da8fdf24df
I20260812 01:01:48.515793 14125 catalog_manager.cc:1552] T 00000000000000000000000000000000 P caf3a11394944625876aeb85be168472: loading cluster ID for follower catalog manager: success
I20260812 01:01:48.518314 14125 catalog_manager.cc:1574] T 00000000000000000000000000000000 P caf3a11394944625876aeb85be168472: acquiring CA information for follower catalog manager: success
I20260812 01:01:48.520207 14125 catalog_manager.cc:1602] T 00000000000000000000000000000000 P caf3a11394944625876aeb85be168472: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 01:01:48.632244 14152 catalog_manager.cc:1259] Loaded cluster ID: 7aafd197861d43d6a59cc7da8fdf24df
I20260812 01:01:48.632359 14152 catalog_manager.cc:1552] T 00000000000000000000000000000000 P afceb7844a9b45a791ea3fce293a43d8: loading cluster ID for follower catalog manager: success
I20260812 01:01:48.634760 14152 catalog_manager.cc:1574] T 00000000000000000000000000000000 P afceb7844a9b45a791ea3fce293a43d8: acquiring CA information for follower catalog manager: success
I20260812 01:01:48.636519 14152 catalog_manager.cc:1602] T 00000000000000000000000000000000 P afceb7844a9b45a791ea3fce293a43d8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 01:01:49.099224 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 13697
I20260812 01:01:49.101135 13807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:01:49.217229 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 13697
I20260812 01:01:49.245951 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 13827
I20260812 01:01:49.247161 13924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:01:49.254580 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:49.253702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:41039 (local address 127.0.0.1:36256)
0812 01:01:49.254280 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:49.254403 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":340,"threads_started":1}
W20260812 01:01:49.254901 13972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.201.126:41039: connect: Connection refused (error 111)
I20260812 01:01:49.256484 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:49.256112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:41039 (local address 127.0.0.1:36270)
0812 01:01:49.256268 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:49.256349 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 01:01:49.256676 13972 consensus_peers.cc:487] T 00000000000000000000000000000000 P 741f68b24e25486c91cd351ba7c5417c -> Peer caf3a11394944625876aeb85be168472 (127.21.201.126:41039): Couldn't send request to peer caf3a11394944625876aeb85be168472. Status: Network error: Client connection negotiation failed: client connection to 127.21.201.126:41039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 01:01:49.361187 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 13827
I20260812 01:01:49.390163 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 13944
I20260812 01:01:49.391404 14052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:01:49.505801 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 13944
2026-08-12T01:01:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5290 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:01:49 dist-test-slave-7f01 krb5kdc[14203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:01:49 dist-test-slave-7f01 krb5kdc[14203](info): set up 2 sockets
Aug 12 01:01:49 dist-test-slave-7f01 krb5kdc[14203](info): commencing operation
krb5kdc: starting...
W20260812 01:01:50.884207 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.328s user 0.003s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:01:51 dist-test-slave-7f01 krb5kdc[14203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:01:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:01:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 01:01:51.176506 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:34811
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:36629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:34811
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:01:51.512894 14232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:01:51.513180 14232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:01:51.513355 14232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:01:51.520539 14232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:01:51.520687 14232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:01:51.520792 14232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:01:51.520901 14232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:01:51.520984 14232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:01:51.530751 14232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:36629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:34811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:34811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:01:51.532101 14232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:01:51.533457 14232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:01:51.545351 14244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:51.545420 14247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:51.545464 14245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:01:51.546708 14232 server_base.cc:1034] running on GCE node
I20260812 01:01:51.547574 14232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:01:51.549079 14232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:01:51.550292 14232 hybrid_clock.cc:648] HybridClock initialized: now 1786496511550232 us; error 53 us; skew 500 ppm
I20260812 01:01:51.550611 14232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:01:51 dist-test-slave-7f01 krb5kdc[14203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496511, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:01:51.558895 14232 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 01:01:51.560565 14232 webserver.cc:460] Webserver started at http://127.21.201.126:37811/ using document root <none> and password file <none>
I20260812 01:01:51.561308 14232 fs_manager.cc:359] Metadata directory not provided
I20260812 01:01:51.561472 14232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:01:51.561800 14232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:01:51.564983 14232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "7098e03d03a547969b8cada6b1a1360b"
format_stamp: "Formatted at 2026-08-12 01:01:51 on dist-test-slave-7f01"
server_key: "38e1a62b8d06e810267b87e38b8012f0"
server_key_iv: "ae50add3e46fb961779fa25dd1aa9a3b"
server_key_version: "encryptionkey@0"
I20260812 01:01:51.565776 14232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "7098e03d03a547969b8cada6b1a1360b"
format_stamp: "Formatted at 2026-08-12 01:01:51 on dist-test-slave-7f01"
server_key: "38e1a62b8d06e810267b87e38b8012f0"
server_key_iv: "ae50add3e46fb961779fa25dd1aa9a3b"
server_key_version: "encryptionkey@0"
I20260812 01:01:51.571796 14232 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 01:01:51.575418 14259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:51.576820 14232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:01:51.577046 14232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "7098e03d03a547969b8cada6b1a1360b"
format_stamp: "Formatted at 2026-08-12 01:01:51 on dist-test-slave-7f01"
server_key: "38e1a62b8d06e810267b87e38b8012f0"
server_key_iv: "ae50add3e46fb961779fa25dd1aa9a3b"
server_key_version: "encryptionkey@0"
I20260812 01:01:51.577265 14232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:01:51.598174 14232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:01:51.602846 14232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:01:51.603161 14232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:01:51.623101 14232 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:34811
I20260812 01:01:51.623137 14343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:34811 every 8 connection(s)
I20260812 01:01:51.624567 14232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:01:51.628489 14347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:01:51.632898 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 14232
I20260812 01:01:51.633150 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:01:51.634009 22309 external_mini_cluster.cc:1428] Setting key 12cb8c01a72cc23a0c51adc9a1aa38da
I20260812 01:01:51.643400 14347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7098e03d03a547969b8cada6b1a1360b: Bootstrap starting.
I20260812 01:01:51.647681 14347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7098e03d03a547969b8cada6b1a1360b: Neither blocks nor log segments found. Creating new log.
I20260812 01:01:51.649223 14347 log.cc:824] T 00000000000000000000000000000000 P 7098e03d03a547969b8cada6b1a1360b: Log is configured to *not* fsync() on all Append() calls
Aug 12 01:01:51 dist-test-slave-7f01 krb5kdc[14203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:01:51.652631 14347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7098e03d03a547969b8cada6b1a1360b: No bootstrap required, opened a new log
I20260812 01:01:51.659667 14353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:51.636119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54288 (local address 127.21.201.126:34811)
0812 01:01:51.636898 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:51.636909 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:51.636951 (+ 42us) server_negotiation.cc:407] Connection header received
0812 01:01:51.638614 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:51.638637 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:51.639028 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:51.639364 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:51.641564 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:51.642910 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:51.645061 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:51.645759 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:51.645922 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:51.651884 (+ 5962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:51.651898 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:51.651912 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:51.651946 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:51.655310 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:51.656341 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:51.656347 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:51.656353 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:51.656531 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:51.657012 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:51.657016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:51.657018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:51.657531 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:51.657768 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:51.658270 (+ 502us) server_negotiation.cc:299] Negotiation successful
0812 01:01:51.658592 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":364,"threads_started":1}
I20260812 01:01:51.662995 14347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7098e03d03a547969b8cada6b1a1360b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7098e03d03a547969b8cada6b1a1360b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 34811 } }
I20260812 01:01:51.663483 14347 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7098e03d03a547969b8cada6b1a1360b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:01:51.663617 14347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7098e03d03a547969b8cada6b1a1360b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7098e03d03a547969b8cada6b1a1360b, State: Initialized, Role: FOLLOWER
I20260812 01:01:51.663985 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:01:51.664316 14347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7098e03d03a547969b8cada6b1a1360b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7098e03d03a547969b8cada6b1a1360b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 34811 } }
I20260812 01:01:51.664580 14347 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7098e03d03a547969b8cada6b1a1360b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:01:51.664733 14347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7098e03d03a547969b8cada6b1a1360b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:01:51.664914 14347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7098e03d03a547969b8cada6b1a1360b [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:01:51.667613 14347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7098e03d03a547969b8cada6b1a1360b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7098e03d03a547969b8cada6b1a1360b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 34811 } }
I20260812 01:01:51.668207 14347 leader_election.cc:302] T 00000000000000000000000000000000 P 7098e03d03a547969b8cada6b1a1360b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7098e03d03a547969b8cada6b1a1360b; no voters:
I20260812 01:01:51.668814 14347 leader_election.cc:288] T 00000000000000000000000000000000 P 7098e03d03a547969b8cada6b1a1360b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:01:51.668918 14357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7098e03d03a547969b8cada6b1a1360b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:01:51.669697 14357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7098e03d03a547969b8cada6b1a1360b [term 1 LEADER]: Becoming Leader. State: Replica: 7098e03d03a547969b8cada6b1a1360b, State: Running, Role: LEADER
I20260812 01:01:51.670235 14357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7098e03d03a547969b8cada6b1a1360b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7098e03d03a547969b8cada6b1a1360b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 34811 } }
I20260812 01:01:51.670995 14347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7098e03d03a547969b8cada6b1a1360b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:01:51.672988 14360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7098e03d03a547969b8cada6b1a1360b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7098e03d03a547969b8cada6b1a1360b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7098e03d03a547969b8cada6b1a1360b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 34811 } } }
I20260812 01:01:51.673501 14360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7098e03d03a547969b8cada6b1a1360b [sys.catalog]: This master's current role is: LEADER
I20260812 01:01:51.675209 14366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:01:51.675190 14361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7098e03d03a547969b8cada6b1a1360b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7098e03d03a547969b8cada6b1a1360b. Latest consensus state: current_term: 1 leader_uuid: "7098e03d03a547969b8cada6b1a1360b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7098e03d03a547969b8cada6b1a1360b" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 34811 } } }
I20260812 01:01:51.675797 14361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7098e03d03a547969b8cada6b1a1360b [sys.catalog]: This master's current role is: LEADER
I20260812 01:01:51.680751 14366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:01:51.691239 14366 catalog_manager.cc:1347] Generated new cluster ID: af5b57599efe4b88b5ead1d4f9c2153c
I20260812 01:01:51.691376 14366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:01:51.707060 14366 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:01:51.708425 14366 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:01:51.715806 14366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7098e03d03a547969b8cada6b1a1360b: Generated new TSK 0
I20260812 01:01:51.716667 14366 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:01:51.725222 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 14232
I20260812 01:01:51.726531 14335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:01:51.821664 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 14232
2026-08-12T01:01:51Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2312 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:01:51 dist-test-slave-7f01 krb5kdc[14409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:01:51 dist-test-slave-7f01 krb5kdc[14409](info): set up 2 sockets
Aug 12 01:01:51 dist-test-slave-7f01 krb5kdc[14409](info): commencing operation
krb5kdc: starting...
W20260812 01:01:53.175611 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.306s user 0.007s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:01:53 dist-test-slave-7f01 krb5kdc[14409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:01:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:01:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 01:01:53.451500 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:46059
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:46059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:01:53.791630 14429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:01:53.792063 14429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:01:53.792254 14429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:01:53.799635 14429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:01:53.799849 14429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:01:53.799960 14429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:01:53.800073 14429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:01:53.800163 14429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:01:53.810496 14429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:46059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:46059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:01:53.812517 14429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:01:53.814031 14429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:01:53.826073 14442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:53.825997 14445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:53.826143 14443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:01:53.827099 14429 server_base.cc:1034] running on GCE node
I20260812 01:01:53.827963 14429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:01:53.829331 14429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:01:53.830564 14429 hybrid_clock.cc:648] HybridClock initialized: now 1786496513830525 us; error 54 us; skew 500 ppm
I20260812 01:01:53.830921 14429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:01:53 dist-test-slave-7f01 krb5kdc[14409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496513, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:01:53.834271 14429 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 01:01:53.835842 14429 webserver.cc:460] Webserver started at http://127.21.201.126:41783/ using document root <none> and password file <none>
I20260812 01:01:53.836539 14429 fs_manager.cc:359] Metadata directory not provided
I20260812 01:01:53.836686 14429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:01:53.836989 14429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:01:53.840392 14429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "86b526280972413ebeff882ad481b790"
format_stamp: "Formatted at 2026-08-12 01:01:53 on dist-test-slave-7f01"
server_key: "af406f1ffed7fbf9f7baf5a90da6c526"
server_key_iv: "65375d5c4608cf8ef067171a16a0f6aa"
server_key_version: "encryptionkey@0"
I20260812 01:01:53.841210 14429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "86b526280972413ebeff882ad481b790"
format_stamp: "Formatted at 2026-08-12 01:01:53 on dist-test-slave-7f01"
server_key: "af406f1ffed7fbf9f7baf5a90da6c526"
server_key_iv: "65375d5c4608cf8ef067171a16a0f6aa"
server_key_version: "encryptionkey@0"
I20260812 01:01:53.846625 14429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 01:01:53.850319 14462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:53.851593 14429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 01:01:53.851856 14429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "86b526280972413ebeff882ad481b790"
format_stamp: "Formatted at 2026-08-12 01:01:53 on dist-test-slave-7f01"
server_key: "af406f1ffed7fbf9f7baf5a90da6c526"
server_key_iv: "65375d5c4608cf8ef067171a16a0f6aa"
server_key_version: "encryptionkey@0"
I20260812 01:01:53.852092 14429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:01:53.880335 14429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:01:53.886395 14429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:01:53.886782 14429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:01:53.906481 14429 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:46059
I20260812 01:01:53.906493 14548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:46059 every 8 connection(s)
I20260812 01:01:53.908012 14429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:01:53.911706 14551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:01:53.916982 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 14429
I20260812 01:01:53.917237 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:01:53.917943 22309 external_mini_cluster.cc:1428] Setting key 856a4535d4fdd1d3dd90df83278cef0c
I20260812 01:01:53.924197 14551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790: Bootstrap starting.
I20260812 01:01:53.928078 14551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790: Neither blocks nor log segments found. Creating new log.
I20260812 01:01:53.929123 14551 log.cc:824] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790: Log is configured to *not* fsync() on all Append() calls
I20260812 01:01:53.932265 14551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790: No bootstrap required, opened a new log
Aug 12 01:01:53 dist-test-slave-7f01 krb5kdc[14409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:01:53.942996 14551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b526280972413ebeff882ad481b790" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46059 } }
I20260812 01:01:53.943483 14551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:01:53.943611 14551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b526280972413ebeff882ad481b790, State: Initialized, Role: FOLLOWER
I20260812 01:01:53.943701 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:53.920132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36748 (local address 127.21.201.126:46059)
0812 01:01:53.921092 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:53.921107 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:53.921161 (+ 54us) server_negotiation.cc:407] Connection header received
0812 01:01:53.923599 (+ 2438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:53.923628 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:53.924162 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:53.924614 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:53.926009 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:53.928085 (+ 2076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:53.929449 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:53.930349 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:53.930546 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:53.934825 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:53.934837 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:53.934862 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:53.934916 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:53.939120 (+ 4204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:53.939852 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:53.939865 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:53.939874 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:53.940084 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:53.940454 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:53.940460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:53.940463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:53.941237 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:53.941579 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:53.941897 (+ 318us) server_negotiation.cc:299] Negotiation successful
0812 01:01:53.942333 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":387,"threads_started":1}
I20260812 01:01:53.944350 14551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b526280972413ebeff882ad481b790" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46059 } }
I20260812 01:01:53.944624 14551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:01:53.944801 14551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:01:53.945010 14551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:01:53.948109 14551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b526280972413ebeff882ad481b790" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46059 } }
I20260812 01:01:53.948554 14551 leader_election.cc:302] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86b526280972413ebeff882ad481b790; no voters:
I20260812 01:01:53.949152 14551 leader_election.cc:288] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:01:53.949265 14567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:01:53.950130 14567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [term 1 LEADER]: Becoming Leader. State: Replica: 86b526280972413ebeff882ad481b790, State: Running, Role: LEADER
I20260812 01:01:53.950809 14567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b526280972413ebeff882ad481b790" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46059 } }
I20260812 01:01:53.951550 14551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:01:53.953976 14568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86b526280972413ebeff882ad481b790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b526280972413ebeff882ad481b790" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46059 } } }
I20260812 01:01:53.953963 14569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86b526280972413ebeff882ad481b790. Latest consensus state: current_term: 1 leader_uuid: "86b526280972413ebeff882ad481b790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b526280972413ebeff882ad481b790" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46059 } } }
I20260812 01:01:53.954387 14568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [sys.catalog]: This master's current role is: LEADER
I20260812 01:01:53.954450 14569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [sys.catalog]: This master's current role is: LEADER
I20260812 01:01:53.955966 14580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:01:53.961194 14580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:01:53.970153 14580 catalog_manager.cc:1347] Generated new cluster ID: 9f307e7a184d412fa2314b73e672dead
I20260812 01:01:53.970266 14580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:01:53.983112 14580 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:01:53.984912 14580 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:01:53.998492 14580 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790: Generated new TSK 0
I20260812 01:01:53.999274 14580 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.201.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
I20260812 01:01:54.076519 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:0
--local_ip_for_outbound_sockets=127.21.201.65
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:46059
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:01:54.423899 14610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:01:54.424219 14610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:01:54.424391 14610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:01:54.431641 14610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:01:54.431823 14610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:01:54.432017 14610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 01:01:54.442085 14610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:46059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:01:54.443727 14610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:01:54.445685 14610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:01:54.460788 14621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:54.460754 14629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:54.460829 14625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:01:54.462070 14610 server_base.cc:1034] running on GCE node
I20260812 01:01:54.462700 14610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:01:54.463642 14610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:01:54.464867 14610 hybrid_clock.cc:648] HybridClock initialized: now 1786496514464810 us; error 55 us; skew 500 ppm
I20260812 01:01:54.465185 14610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:01:54 dist-test-slave-7f01 krb5kdc[14409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496514, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:01:54.468390 14610 init.cc:377] Logged in from keytab as kudu/127.21.201.65@KRBTEST.COM (short username kudu)
I20260812 01:01:54.469928 14610 webserver.cc:460] Webserver started at http://127.21.201.65:34645/ using document root <none> and password file <none>
I20260812 01:01:54.470615 14610 fs_manager.cc:359] Metadata directory not provided
I20260812 01:01:54.470769 14610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:01:54.471086 14610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:01:54.474076 14610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/data/instance:
uuid: "a300244bb80847f68991901ff6ed4f00"
format_stamp: "Formatted at 2026-08-12 01:01:54 on dist-test-slave-7f01"
server_key: "dd08d3277f3e90779b064b315e917363"
server_key_iv: "1d4825eee7ddf41a0f3d17287fdd13b4"
server_key_version: "encryptionkey@0"
I20260812 01:01:54.474869 14610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance:
uuid: "a300244bb80847f68991901ff6ed4f00"
format_stamp: "Formatted at 2026-08-12 01:01:54 on dist-test-slave-7f01"
server_key: "dd08d3277f3e90779b064b315e917363"
server_key_iv: "1d4825eee7ddf41a0f3d17287fdd13b4"
server_key_version: "encryptionkey@0"
I20260812 01:01:54.479998 14610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 01:01:54.483316 14645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:54.484611 14610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:01:54.484802 14610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "a300244bb80847f68991901ff6ed4f00"
format_stamp: "Formatted at 2026-08-12 01:01:54 on dist-test-slave-7f01"
server_key: "dd08d3277f3e90779b064b315e917363"
server_key_iv: "1d4825eee7ddf41a0f3d17287fdd13b4"
server_key_version: "encryptionkey@0"
I20260812 01:01:54.485029 14610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:01:54.508688 14610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:01:54.512802 14610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:01:54.513104 14610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:01:54.514102 14610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:01:54.515452 14610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:01:54.515559 14610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:54.515705 14610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:01:54.515810 14610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 01:01:54.550871 14610 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:35855
I20260812 01:01:54.550932 14844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:35855 every 8 connection(s)
I20260812 01:01:54.552434 14610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 01:01:54.558246 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 14610
I20260812 01:01:54.558483 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance
I20260812 01:01:54.559332 22309 external_mini_cluster.cc:1428] Setting key f722f90d5514ba5db12c611b74bb5949
Aug 12 01:01:54 dist-test-slave-7f01 krb5kdc[14409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496514, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:01:54.575626 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:54.555180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:48469 (local address 127.21.201.126:46059)
0812 01:01:54.555832 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:54.555836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:54.556833 (+ 997us) server_negotiation.cc:407] Connection header received
0812 01:01:54.559378 (+ 2545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:54.559383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:54.559497 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:54.559605 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:54.561482 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:54.562753 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:54.564423 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:54.564851 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:54.564996 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:54.568821 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:54.568829 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:54.568836 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:54.568880 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:54.572354 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:54.573143 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:54.573146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:54.573149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:54.573267 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:54.573844 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:54.573849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:54.573851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:54.574120 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:54.574247 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:54.575172 (+ 925us) server_negotiation.cc:299] Negotiation successful
0812 01:01:54.575355 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":314,"threads_started":1}
I20260812 01:01:54.577215 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:54.555715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.65:48469)
0812 01:01:54.556586 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:54.556625 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:54.559055 (+ 2430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:54.559904 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:54.559915 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:54.560501 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:01:54.561291 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:54.561318 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:54.562919 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:54.562925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:54.564240 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:54.564252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:54.565091 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:54.565100 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:54.565338 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:54.566065 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:01:54.566088 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:01:54.568576 (+ 2488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:01:54.572517 (+ 3941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:54.572528 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:54.572548 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:54.573008 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:54.573399 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:54.573406 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:54.573411 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:54.573721 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:54.574273 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:01:54.574283 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:01:54.574744 (+ 461us) client_negotiation.cc:769] Sending connection context
0812 01:01:54.575060 (+ 316us) client_negotiation.cc:240] Negotiation successful
0812 01:01:54.575645 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":306,"threads_started":1}
I20260812 01:01:54.578828 14845 heartbeater.cc:347] Connected to a master server at 127.21.201.126:46059
I20260812 01:01:54.579167 14845 heartbeater.cc:464] Registering TS with master...
I20260812 01:01:54.580008 14845 heartbeater.cc:511] Master 127.21.201.126:46059 requested a full tablet report, sending...
I20260812 01:01:54.582139 14489 ts_manager.cc:194] Registered new tserver with Master: a300244bb80847f68991901ff6ed4f00 (127.21.201.65:35855)
I20260812 01:01:54.583626 14489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.65@KRBTEST.COM'} at 127.21.201.65:48469
WARNING: no policy specified for kudu/127.21.201.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
I20260812 01:01:54.623266 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:0
--local_ip_for_outbound_sockets=127.21.201.66
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:46059
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:01:54.965500 14856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:01:54.965787 14856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:01:54.965900 14856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:01:54.973356 14856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:01:54.973470 14856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:01:54.973598 14856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 01:01:54.983445 14856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:46059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:01:54.985037 14856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:01:54.986546 14856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:01:55.000746 14871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:55.000804 14876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:55.000830 14866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:01:55.001708 14856 server_base.cc:1034] running on GCE node
I20260812 01:01:55.002324 14856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:01:55.003266 14856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:01:55.004496 14856 hybrid_clock.cc:648] HybridClock initialized: now 1786496515004459 us; error 56 us; skew 500 ppm
I20260812 01:01:55.004822 14856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:01:55 dist-test-slave-7f01 krb5kdc[14409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496515, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:01:55.008144 14856 init.cc:377] Logged in from keytab as kudu/127.21.201.66@KRBTEST.COM (short username kudu)
I20260812 01:01:55.009581 14856 webserver.cc:460] Webserver started at http://127.21.201.66:36363/ using document root <none> and password file <none>
I20260812 01:01:55.010296 14856 fs_manager.cc:359] Metadata directory not provided
I20260812 01:01:55.010469 14856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:01:55.010793 14856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:01:55.013846 14856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/data/instance:
uuid: "7e8da74e740d4510a457fb5e6dd1d819"
format_stamp: "Formatted at 2026-08-12 01:01:55 on dist-test-slave-7f01"
server_key: "1f2696dfc83d5691f9e1af5b3585ba38"
server_key_iv: "f8e645c442be961a09a5b5c1b6bf1b32"
server_key_version: "encryptionkey@0"
I20260812 01:01:55.014681 14856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance:
uuid: "7e8da74e740d4510a457fb5e6dd1d819"
format_stamp: "Formatted at 2026-08-12 01:01:55 on dist-test-slave-7f01"
server_key: "1f2696dfc83d5691f9e1af5b3585ba38"
server_key_iv: "f8e645c442be961a09a5b5c1b6bf1b32"
server_key_version: "encryptionkey@0"
I20260812 01:01:55.020169 14856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260812 01:01:55.023646 14889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:55.024984 14856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:01:55.025177 14856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "7e8da74e740d4510a457fb5e6dd1d819"
format_stamp: "Formatted at 2026-08-12 01:01:55 on dist-test-slave-7f01"
server_key: "1f2696dfc83d5691f9e1af5b3585ba38"
server_key_iv: "f8e645c442be961a09a5b5c1b6bf1b32"
server_key_version: "encryptionkey@0"
I20260812 01:01:55.025393 14856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:01:55.042526 14856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:01:55.046492 14856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:01:55.046824 14856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:01:55.047845 14856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:01:55.049248 14856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:01:55.049361 14856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:55.049536 14856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:01:55.049618 14856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:55.085152 14856 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:34007
I20260812 01:01:55.085192 15067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:34007 every 8 connection(s)
I20260812 01:01:55.086480 14856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 01:01:55.090806 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 14856
I20260812 01:01:55.091024 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance
I20260812 01:01:55.091658 22309 external_mini_cluster.cc:1428] Setting key 350cbcf5e2177cbbd3cb85711faf9012
Aug 12 01:01:55 dist-test-slave-7f01 krb5kdc[14409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496515, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:01:55.110105 15074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:55.089209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:36005 (local address 127.21.201.126:46059)
0812 01:01:55.089728 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:55.089732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:55.090753 (+ 1021us) server_negotiation.cc:407] Connection header received
0812 01:01:55.093359 (+ 2606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:55.093364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:55.093468 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:55.093590 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:55.095663 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.096466 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.098926 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.099363 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.099468 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:55.103870 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:55.103876 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:55.103880 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:55.103914 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:55.106841 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:55.107615 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:55.107621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:55.107631 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:55.107820 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:55.108270 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:55.108276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:55.108280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:55.108569 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:55.108724 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:55.109418 (+ 694us) server_negotiation.cc:299] Negotiation successful
0812 01:01:55.109673 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":274,"threads_started":1}
I20260812 01:01:55.111119 15076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:55.089730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.66:36005)
0812 01:01:55.090541 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:55.090604 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:55.093081 (+ 2477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:55.093844 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:55.093852 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:55.094635 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:01:55.095469 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:55.095496 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.096639 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.096647 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:55.098777 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:55.098795 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.099596 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.099605 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:55.099969 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:55.100743 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:01:55.100780 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:01:55.103583 (+ 2803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:01:55.107004 (+ 3421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:55.107013 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:55.107030 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:55.107474 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:55.107941 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:55.107946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:55.107950 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:55.108150 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:55.108711 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:01:55.108719 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:01:55.109068 (+ 349us) client_negotiation.cc:769] Sending connection context
0812 01:01:55.109351 (+ 283us) client_negotiation.cc:240] Negotiation successful
0812 01:01:55.109904 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":288,"threads_started":1}
I20260812 01:01:55.112880 15069 heartbeater.cc:347] Connected to a master server at 127.21.201.126:46059
I20260812 01:01:55.113194 15069 heartbeater.cc:464] Registering TS with master...
I20260812 01:01:55.113950 15069 heartbeater.cc:511] Master 127.21.201.126:46059 requested a full tablet report, sending...
I20260812 01:01:55.115487 14489 ts_manager.cc:194] Registered new tserver with Master: 7e8da74e740d4510a457fb5e6dd1d819 (127.21.201.66:34007)
I20260812 01:01:55.116384 14489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.66@KRBTEST.COM'} at 127.21.201.66:36005
WARNING: no policy specified for kudu/127.21.201.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
I20260812 01:01:55.159212 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:0
--local_ip_for_outbound_sockets=127.21.201.67
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:46059
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:01:55.511471 15082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:01:55.511849 15082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:01:55.512063 15082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:01:55.519538 15082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:01:55.519649 15082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:01:55.519884 15082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 01:01:55.529951 15082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:46059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:01:55.531247 15082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:01:55.532644 15082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:01:55.546648 15092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:55.546692 15089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:55.546661 15095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:01:55.547674 15082 server_base.cc:1034] running on GCE node
I20260812 01:01:55.548372 15082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:01:55.549301 15082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:01:55.550525 15082 hybrid_clock.cc:648] HybridClock initialized: now 1786496515550495 us; error 47 us; skew 500 ppm
I20260812 01:01:55.550853 15082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:01:55 dist-test-slave-7f01 krb5kdc[14409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496515, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:01:55.554441 15082 init.cc:377] Logged in from keytab as kudu/127.21.201.67@KRBTEST.COM (short username kudu)
I20260812 01:01:55.556795 15082 webserver.cc:460] Webserver started at http://127.21.201.67:38743/ using document root <none> and password file <none>
I20260812 01:01:55.557546 15082 fs_manager.cc:359] Metadata directory not provided
I20260812 01:01:55.557688 15082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:01:55.558017 15082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:01:55.560891 15082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/data/instance:
uuid: "430435c9b2314d60b15673a369a0c5a3"
format_stamp: "Formatted at 2026-08-12 01:01:55 on dist-test-slave-7f01"
server_key: "a328c36199471f699ec4562866f8f5aa"
server_key_iv: "16e8525042db1c138c59b8a98926fc4e"
server_key_version: "encryptionkey@0"
I20260812 01:01:55.561686 15082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance:
uuid: "430435c9b2314d60b15673a369a0c5a3"
format_stamp: "Formatted at 2026-08-12 01:01:55 on dist-test-slave-7f01"
server_key: "a328c36199471f699ec4562866f8f5aa"
server_key_iv: "16e8525042db1c138c59b8a98926fc4e"
server_key_version: "encryptionkey@0"
I20260812 01:01:55.566799 15082 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260812 01:01:55.570509 15113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:55.571949 15082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:01:55.572115 15082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "430435c9b2314d60b15673a369a0c5a3"
format_stamp: "Formatted at 2026-08-12 01:01:55 on dist-test-slave-7f01"
server_key: "a328c36199471f699ec4562866f8f5aa"
server_key_iv: "16e8525042db1c138c59b8a98926fc4e"
server_key_version: "encryptionkey@0"
I20260812 01:01:55.572336 15082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:01:55.585352 15082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:01:55.586530 14845 heartbeater.cc:503] Master 127.21.201.126:46059 was elected leader, sending a full tablet report...
I20260812 01:01:55.589897 15082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:01:55.590188 15082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:01:55.591125 15082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:01:55.592506 15082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:01:55.592581 15082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:55.592756 15082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:01:55.592815 15082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:55.628621 15082 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:34275
I20260812 01:01:55.628734 15301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:34275 every 8 connection(s)
I20260812 01:01:55.630084 15082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 01:01:55.636476 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 15082
I20260812 01:01:55.636716 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance
I20260812 01:01:55.637215 22309 external_mini_cluster.cc:1428] Setting key 8902e94bb36d3543b4ee7c024cd2df80
Aug 12 01:01:55 dist-test-slave-7f01 krb5kdc[14409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496515, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:01:55.652436 15305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:55.632715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:45551 (local address 127.21.201.126:46059)
0812 01:01:55.633260 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:55.633266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:55.634075 (+ 809us) server_negotiation.cc:407] Connection header received
0812 01:01:55.635973 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:55.635979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:55.636087 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:55.636212 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:55.638067 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.639079 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.641041 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.641458 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.641600 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:55.645478 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:55.645483 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:55.645487 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:55.645523 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:55.648823 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:55.649691 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:55.649695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:55.649698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:55.649837 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:55.650320 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:55.650330 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:55.650332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:55.650628 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:55.650817 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:55.651603 (+ 786us) server_negotiation.cc:299] Negotiation successful
0812 01:01:55.651938 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":293,"threads_started":1}
I20260812 01:01:55.653558 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:55.633150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.67:45551)
0812 01:01:55.633913 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:55.633952 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:55.635638 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:55.636457 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:55.636465 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:55.637107 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:01:55.637799 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:55.637828 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.639227 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.639233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:55.640890 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:55.640903 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.641695 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.641703 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:55.641946 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:55.642681 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:01:55.642705 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:01:55.645234 (+ 2529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:01:55.649009 (+ 3775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:55.649019 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:55.649048 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:55.649539 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:55.649973 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:55.649978 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:55.649982 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:55.650188 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:55.650772 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:01:55.650780 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:01:55.651175 (+ 395us) client_negotiation.cc:769] Sending connection context
0812 01:01:55.651483 (+ 308us) client_negotiation.cc:240] Negotiation successful
0812 01:01:55.652141 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":320,"threads_started":1}
I20260812 01:01:55.655648 15303 heartbeater.cc:347] Connected to a master server at 127.21.201.126:46059
I20260812 01:01:55.656503 15303 heartbeater.cc:464] Registering TS with master...
I20260812 01:01:55.657666 15303 heartbeater.cc:511] Master 127.21.201.126:46059 requested a full tablet report, sending...
I20260812 01:01:55.659591 14489 ts_manager.cc:194] Registered new tserver with Master: 430435c9b2314d60b15673a369a0c5a3 (127.21.201.67:34275)
I20260812 01:01:55.660496 14489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.67@KRBTEST.COM'} at 127.21.201.67:45551
I20260812 01:01:55.667522 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:01:55.686187 15305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:55.673155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36752 (local address 127.21.201.126:46059)
0812 01:01:55.673401 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:55.673410 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:55.673862 (+ 452us) server_negotiation.cc:407] Connection header received
0812 01:01:55.674026 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:55.674030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:55.674109 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:55.674202 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:55.675683 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.676571 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.677952 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.678354 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.678443 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:55.680875 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:55.680880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:55.680883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:55.680914 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:55.683672 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:55.684346 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:55.684350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:55.684352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:55.684482 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:55.684818 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:55.684821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:55.684823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:55.685160 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:55.685348 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:55.685634 (+ 286us) server_negotiation.cc:299] Negotiation successful
0812 01:01:55.685806 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 01:01:55.695242 14489 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:01:55.704442 14489 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:01:55.732550 15329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:55.718330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41344 (local address 127.21.201.65:35855)
0812 01:01:55.719105 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:55.719114 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:55.719592 (+ 478us) server_negotiation.cc:407] Connection header received
0812 01:01:55.720144 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:55.720153 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:55.720440 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:55.720692 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:01:55.721992 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.724109 (+ 2117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.727985 (+ 3876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.730169 (+ 2184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.730321 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:55.730638 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:55.731707 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0812 01:01:55.732011 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":392,"threads_started":1}
I20260812 01:01:55.733683 15327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:55.718147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:35855 (local address 127.0.0.1:41344)
0812 01:01:55.719448 (+ 1301us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:55.719485 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:55.719802 (+ 317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:55.721051 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:55.721063 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:55.721119 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:01:55.721687 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:55.721704 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.724352 (+ 2648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.724361 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:55.727714 (+ 3353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:55.727752 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.731015 (+ 3263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.731025 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:55.731336 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:55.731409 (+ 73us) client_negotiation.cc:769] Sending connection context
0812 01:01:55.732856 (+ 1447us) client_negotiation.cc:240] Negotiation successful
0812 01:01:55.733154 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1064,"thread_start_us":927,"threads_started":1}
I20260812 01:01:55.736132 15330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:55.718646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:34275 (local address 127.0.0.1:55966)
0812 01:01:55.720100 (+ 1454us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:55.720121 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:55.720238 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:55.722409 (+ 2171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:55.722417 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:55.722450 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:01:55.723030 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:55.723046 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.727280 (+ 4234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.727289 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:55.731679 (+ 4390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:55.731705 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.734846 (+ 3141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.734852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:55.735116 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:55.735172 (+ 56us) client_negotiation.cc:769] Sending connection context
0812 01:01:55.735293 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 01:01:55.735593 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1316,"thread_start_us":1141,"threads_started":1}
I20260812 01:01:55.736457 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:55.719217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55966 (local address 127.21.201.67:34275)
0812 01:01:55.719903 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:55.719914 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:55.720355 (+ 441us) server_negotiation.cc:407] Connection header received
0812 01:01:55.720463 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:55.720470 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:55.720704 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:55.720922 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:01:55.723531 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.727063 (+ 3532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.731973 (+ 4910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.733816 (+ 1843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.733958 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:55.734202 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:55.735889 (+ 1687us) server_negotiation.cc:299] Negotiation successful
0812 01:01:55.736092 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":341,"threads_started":1}
I20260812 01:01:55.738070 14732 tablet_service.cc:1467] Processing CreateTablet for tablet 642eb11d79104b79b5da8998c9fd48d5 (DEFAULT_TABLE table=test-table [id=4de23b9c6f574c1bacf7ee6cf5000159]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:01:55.739749 14732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 642eb11d79104b79b5da8998c9fd48d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:01:55.741039 15189 tablet_service.cc:1467] Processing CreateTablet for tablet 642eb11d79104b79b5da8998c9fd48d5 (DEFAULT_TABLE table=test-table [id=4de23b9c6f574c1bacf7ee6cf5000159]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:01:55.742400 15189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 642eb11d79104b79b5da8998c9fd48d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:01:55.743852 15327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:55.720175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:34007 (local address 127.0.0.1:52792)
0812 01:01:55.734396 (+ 14221us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:55.734425 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:55.734537 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:55.737273 (+ 2736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:55.737278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:55.737306 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:01:55.737456 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:55.737466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.739212 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.739220 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:55.741186 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:55.741202 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.743059 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.743067 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:55.743260 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:55.743296 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 01:01:55.743370 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 01:01:55.743513 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14090,"thread_start_us":14359,"threads_started":1}
I20260812 01:01:55.744038 15333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:55.720488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52792 (local address 127.21.201.66:34007)
0812 01:01:55.736768 (+ 16280us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:55.736777 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:55.736807 (+ 30us) server_negotiation.cc:407] Connection header received
0812 01:01:55.736907 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:55.736912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:55.737084 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:55.737237 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:01:55.737639 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.739047 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.741376 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.742855 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.743002 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:55.743183 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:55.743484 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 01:01:55.743660 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16117,"thread_start_us":16119,"threads_started":1}
I20260812 01:01:55.751925 15334 tablet_bootstrap.cc:492] T 642eb11d79104b79b5da8998c9fd48d5 P a300244bb80847f68991901ff6ed4f00: Bootstrap starting.
I20260812 01:01:55.751554 14973 tablet_service.cc:1467] Processing CreateTablet for tablet 642eb11d79104b79b5da8998c9fd48d5 (DEFAULT_TABLE table=test-table [id=4de23b9c6f574c1bacf7ee6cf5000159]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:01:55.753036 14973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 642eb11d79104b79b5da8998c9fd48d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:01:55.755255 15334 tablet_bootstrap.cc:654] T 642eb11d79104b79b5da8998c9fd48d5 P a300244bb80847f68991901ff6ed4f00: Neither blocks nor log segments found. Creating new log.
I20260812 01:01:55.755765 15336 tablet_bootstrap.cc:492] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3: Bootstrap starting.
I20260812 01:01:55.756780 15334 log.cc:824] T 642eb11d79104b79b5da8998c9fd48d5 P a300244bb80847f68991901ff6ed4f00: Log is configured to *not* fsync() on all Append() calls
I20260812 01:01:55.759584 15336 tablet_bootstrap.cc:654] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3: Neither blocks nor log segments found. Creating new log.
I20260812 01:01:55.760236 15334 tablet_bootstrap.cc:492] T 642eb11d79104b79b5da8998c9fd48d5 P a300244bb80847f68991901ff6ed4f00: No bootstrap required, opened a new log
I20260812 01:01:55.760519 15334 ts_tablet_manager.cc:1397] T 642eb11d79104b79b5da8998c9fd48d5 P a300244bb80847f68991901ff6ed4f00: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260812 01:01:55.760900 15336 log.cc:824] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3: Log is configured to *not* fsync() on all Append() calls
I20260812 01:01:55.764778 15336 tablet_bootstrap.cc:492] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3: No bootstrap required, opened a new log
I20260812 01:01:55.765053 15336 ts_tablet_manager.cc:1397] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260812 01:01:55.767155 15342 tablet_bootstrap.cc:492] T 642eb11d79104b79b5da8998c9fd48d5 P 7e8da74e740d4510a457fb5e6dd1d819: Bootstrap starting.
I20260812 01:01:55.770509 15342 tablet_bootstrap.cc:654] T 642eb11d79104b79b5da8998c9fd48d5 P 7e8da74e740d4510a457fb5e6dd1d819: Neither blocks nor log segments found. Creating new log.
I20260812 01:01:55.771354 15334 raft_consensus.cc:348] T 642eb11d79104b79b5da8998c9fd48d5 P a300244bb80847f68991901ff6ed4f00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a300244bb80847f68991901ff6ed4f00" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35855 } } peers { permanent_uuid: "430435c9b2314d60b15673a369a0c5a3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34275 } } peers { permanent_uuid: "7e8da74e740d4510a457fb5e6dd1d819" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34007 } }
I20260812 01:01:55.771641 15342 log.cc:824] T 642eb11d79104b79b5da8998c9fd48d5 P 7e8da74e740d4510a457fb5e6dd1d819: Log is configured to *not* fsync() on all Append() calls
I20260812 01:01:55.771798 15334 raft_consensus.cc:374] T 642eb11d79104b79b5da8998c9fd48d5 P a300244bb80847f68991901ff6ed4f00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:01:55.771908 15334 raft_consensus.cc:729] T 642eb11d79104b79b5da8998c9fd48d5 P a300244bb80847f68991901ff6ed4f00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a300244bb80847f68991901ff6ed4f00, State: Initialized, Role: FOLLOWER
I20260812 01:01:55.772523 15334 consensus_queue.cc:260] T 642eb11d79104b79b5da8998c9fd48d5 P a300244bb80847f68991901ff6ed4f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a300244bb80847f68991901ff6ed4f00" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35855 } } peers { permanent_uuid: "430435c9b2314d60b15673a369a0c5a3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34275 } } peers { permanent_uuid: "7e8da74e740d4510a457fb5e6dd1d819" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34007 } }
I20260812 01:01:55.774380 15334 ts_tablet_manager.cc:1428] T 642eb11d79104b79b5da8998c9fd48d5 P a300244bb80847f68991901ff6ed4f00: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260812 01:01:55.775772 15336 raft_consensus.cc:348] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a300244bb80847f68991901ff6ed4f00" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35855 } } peers { permanent_uuid: "430435c9b2314d60b15673a369a0c5a3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34275 } } peers { permanent_uuid: "7e8da74e740d4510a457fb5e6dd1d819" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34007 } }
I20260812 01:01:55.776238 15336 raft_consensus.cc:374] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:01:55.776347 15336 raft_consensus.cc:729] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 430435c9b2314d60b15673a369a0c5a3, State: Initialized, Role: FOLLOWER
I20260812 01:01:55.776542 15342 tablet_bootstrap.cc:492] T 642eb11d79104b79b5da8998c9fd48d5 P 7e8da74e740d4510a457fb5e6dd1d819: No bootstrap required, opened a new log
I20260812 01:01:55.776813 15342 ts_tablet_manager.cc:1397] T 642eb11d79104b79b5da8998c9fd48d5 P 7e8da74e740d4510a457fb5e6dd1d819: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260812 01:01:55.776948 15336 consensus_queue.cc:260] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a300244bb80847f68991901ff6ed4f00" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35855 } } peers { permanent_uuid: "430435c9b2314d60b15673a369a0c5a3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34275 } } peers { permanent_uuid: "7e8da74e740d4510a457fb5e6dd1d819" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34007 } }
I20260812 01:01:55.780401 15303 heartbeater.cc:503] Master 127.21.201.126:46059 was elected leader, sending a full tablet report...
I20260812 01:01:55.781103 15336 ts_tablet_manager.cc:1428] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3: Time spent starting tablet: real 0.016s user 0.009s sys 0.007s
I20260812 01:01:55.786060 15342 raft_consensus.cc:348] T 642eb11d79104b79b5da8998c9fd48d5 P 7e8da74e740d4510a457fb5e6dd1d819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a300244bb80847f68991901ff6ed4f00" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35855 } } peers { permanent_uuid: "430435c9b2314d60b15673a369a0c5a3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34275 } } peers { permanent_uuid: "7e8da74e740d4510a457fb5e6dd1d819" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34007 } }
I20260812 01:01:55.786379 15342 raft_consensus.cc:374] T 642eb11d79104b79b5da8998c9fd48d5 P 7e8da74e740d4510a457fb5e6dd1d819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:01:55.786454 15342 raft_consensus.cc:729] T 642eb11d79104b79b5da8998c9fd48d5 P 7e8da74e740d4510a457fb5e6dd1d819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e8da74e740d4510a457fb5e6dd1d819, State: Initialized, Role: FOLLOWER
I20260812 01:01:55.786882 15342 consensus_queue.cc:260] T 642eb11d79104b79b5da8998c9fd48d5 P 7e8da74e740d4510a457fb5e6dd1d819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a300244bb80847f68991901ff6ed4f00" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35855 } } peers { permanent_uuid: "430435c9b2314d60b15673a369a0c5a3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34275 } } peers { permanent_uuid: "7e8da74e740d4510a457fb5e6dd1d819" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34007 } }
I20260812 01:01:55.787858 15069 heartbeater.cc:503] Master 127.21.201.126:46059 was elected leader, sending a full tablet report...
I20260812 01:01:55.788266 15342 ts_tablet_manager.cc:1428] T 642eb11d79104b79b5da8998c9fd48d5 P 7e8da74e740d4510a457fb5e6dd1d819: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
W20260812 01:01:55.804960 14847 tablet.cc:2369] T 642eb11d79104b79b5da8998c9fd48d5 P a300244bb80847f68991901ff6ed4f00: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:01:55.838651 15072 tablet.cc:2369] T 642eb11d79104b79b5da8998c9fd48d5 P 7e8da74e740d4510a457fb5e6dd1d819: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:01:55.881945 15304 tablet.cc:2369] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:01:55.975150 15346 raft_consensus.cc:482] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:01:55.975631 15346 raft_consensus.cc:504] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a300244bb80847f68991901ff6ed4f00" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35855 } } peers { permanent_uuid: "430435c9b2314d60b15673a369a0c5a3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34275 } } peers { permanent_uuid: "7e8da74e740d4510a457fb5e6dd1d819" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34007 } }
I20260812 01:01:55.977560 15346 leader_election.cc:288] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a300244bb80847f68991901ff6ed4f00 (127.21.201.65:35855), 7e8da74e740d4510a457fb5e6dd1d819 (127.21.201.66:34007)
I20260812 01:01:55.983654 15329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:55.978096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:50515 (local address 127.21.201.65:35855)
0812 01:01:55.978317 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:55.978324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:55.978604 (+ 280us) server_negotiation.cc:407] Connection header received
0812 01:01:55.978843 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:55.978847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:55.978913 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:55.979023 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:01:55.979447 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.980265 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.981883 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.982697 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.982878 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:55.982955 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:55.983235 (+ 280us) server_negotiation.cc:299] Negotiation successful
0812 01:01:55.983371 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 01:01:55.983654 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:55.978174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:35855 (local address 127.21.201.67:50515)
0812 01:01:55.978488 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:55.978505 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:55.978680 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:55.979049 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:55.979054 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:55.979081 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:01:55.979291 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:55.979301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.980396 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.980403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:55.981709 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:55.981723 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.982868 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.982885 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:55.983077 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:55.983102 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 01:01:55.983207 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 01:01:55.983347 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":1}
I20260812 01:01:55.984570 15333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:55.978097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:46599 (local address 127.21.201.66:34007)
0812 01:01:55.978317 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:55.978324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:55.979167 (+ 843us) server_negotiation.cc:407] Connection header received
0812 01:01:55.979489 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:55.979496 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:55.979569 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:55.979685 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:01:55.980314 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.981064 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.982770 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.983572 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.983786 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:55.983863 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:55.984152 (+ 289us) server_negotiation.cc:299] Negotiation successful
0812 01:01:55.984289 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 01:01:55.984653 15349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:55.978181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:34007 (local address 127.21.201.67:46599)
0812 01:01:55.979043 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:55.979058 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:55.979223 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:55.979701 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:55.979706 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:55.979731 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:01:55.980124 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:55.980134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.981225 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.981231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:55.982604 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:55.982615 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:55.983808 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:55.983814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:55.983987 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:55.984012 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 01:01:55.984121 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 01:01:55.984256 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":43,"thread_start_us":486,"threads_started":1}
I20260812 01:01:55.985467 14771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642eb11d79104b79b5da8998c9fd48d5" candidate_uuid: "430435c9b2314d60b15673a369a0c5a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a300244bb80847f68991901ff6ed4f00" is_pre_election: true
I20260812 01:01:55.985656 15001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642eb11d79104b79b5da8998c9fd48d5" candidate_uuid: "430435c9b2314d60b15673a369a0c5a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e8da74e740d4510a457fb5e6dd1d819" is_pre_election: true
I20260812 01:01:55.986032 14771 raft_consensus.cc:2454] T 642eb11d79104b79b5da8998c9fd48d5 P a300244bb80847f68991901ff6ed4f00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 430435c9b2314d60b15673a369a0c5a3 in term 0.
I20260812 01:01:55.986327 15001 raft_consensus.cc:2454] T 642eb11d79104b79b5da8998c9fd48d5 P 7e8da74e740d4510a457fb5e6dd1d819 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 430435c9b2314d60b15673a369a0c5a3 in term 0.
I20260812 01:01:55.986881 15124 leader_election.cc:302] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 430435c9b2314d60b15673a369a0c5a3, a300244bb80847f68991901ff6ed4f00; no voters:
I20260812 01:01:55.987600 15346 raft_consensus.cc:2781] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:01:55.987792 15346 raft_consensus.cc:482] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:01:55.987929 15346 raft_consensus.cc:3037] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:01:55.990516 15346 raft_consensus.cc:504] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a300244bb80847f68991901ff6ed4f00" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35855 } } peers { permanent_uuid: "430435c9b2314d60b15673a369a0c5a3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34275 } } peers { permanent_uuid: "7e8da74e740d4510a457fb5e6dd1d819" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34007 } }
I20260812 01:01:55.991205 15346 leader_election.cc:288] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3 [CANDIDATE]: Term 1 election: Requested vote from peers a300244bb80847f68991901ff6ed4f00 (127.21.201.65:35855), 7e8da74e740d4510a457fb5e6dd1d819 (127.21.201.66:34007)
I20260812 01:01:55.991815 15001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642eb11d79104b79b5da8998c9fd48d5" candidate_uuid: "430435c9b2314d60b15673a369a0c5a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e8da74e740d4510a457fb5e6dd1d819"
I20260812 01:01:55.991815 14771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642eb11d79104b79b5da8998c9fd48d5" candidate_uuid: "430435c9b2314d60b15673a369a0c5a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a300244bb80847f68991901ff6ed4f00"
I20260812 01:01:55.992058 15001 raft_consensus.cc:3037] T 642eb11d79104b79b5da8998c9fd48d5 P 7e8da74e740d4510a457fb5e6dd1d819 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:01:55.992089 14771 raft_consensus.cc:3037] T 642eb11d79104b79b5da8998c9fd48d5 P a300244bb80847f68991901ff6ed4f00 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:01:55.996269 14771 raft_consensus.cc:2454] T 642eb11d79104b79b5da8998c9fd48d5 P a300244bb80847f68991901ff6ed4f00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 430435c9b2314d60b15673a369a0c5a3 in term 1.
I20260812 01:01:55.996269 15001 raft_consensus.cc:2454] T 642eb11d79104b79b5da8998c9fd48d5 P 7e8da74e740d4510a457fb5e6dd1d819 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 430435c9b2314d60b15673a369a0c5a3 in term 1.
I20260812 01:01:55.996843 15118 leader_election.cc:302] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 430435c9b2314d60b15673a369a0c5a3, 7e8da74e740d4510a457fb5e6dd1d819; no voters:
I20260812 01:01:55.997416 15346 raft_consensus.cc:2781] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:01:55.998157 15346 raft_consensus.cc:686] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3 [term 1 LEADER]: Becoming Leader. State: Replica: 430435c9b2314d60b15673a369a0c5a3, State: Running, Role: LEADER
I20260812 01:01:55.998960 15346 consensus_queue.cc:237] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a300244bb80847f68991901ff6ed4f00" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35855 } } peers { permanent_uuid: "430435c9b2314d60b15673a369a0c5a3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34275 } } peers { permanent_uuid: "7e8da74e740d4510a457fb5e6dd1d819" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34007 } }
I20260812 01:01:56.004276 14488 catalog_manager.cc:5495] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 430435c9b2314d60b15673a369a0c5a3 (127.21.201.67). New cstate: current_term: 1 leader_uuid: "430435c9b2314d60b15673a369a0c5a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a300244bb80847f68991901ff6ed4f00" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "430435c9b2314d60b15673a369a0c5a3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e8da74e740d4510a457fb5e6dd1d819" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34007 } health_report { overall_health: UNKNOWN } } }
I20260812 01:01:56.042955 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:56.037313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55976 (local address 127.21.201.67:34275)
0812 01:01:56.037550 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:56.037556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:56.037580 (+ 24us) server_negotiation.cc:407] Connection header received
0812 01:01:56.037707 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:56.037712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:56.037798 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:56.037963 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:01:56.038329 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:56.039136 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:56.040703 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:56.041127 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:56.041266 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:56.041796 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:01:56.042330 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:01:56.042512 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:56.042566 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 01:01:56.042657 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 01:01:56.053761 15001 raft_consensus.cc:1261] T 642eb11d79104b79b5da8998c9fd48d5 P 7e8da74e740d4510a457fb5e6dd1d819 [term 1 FOLLOWER]: Refusing update from remote peer 430435c9b2314d60b15673a369a0c5a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:01:56.053944 14771 raft_consensus.cc:1261] T 642eb11d79104b79b5da8998c9fd48d5 P a300244bb80847f68991901ff6ed4f00 [term 1 FOLLOWER]: Refusing update from remote peer 430435c9b2314d60b15673a369a0c5a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:01:56.055011 15353 consensus_queue.cc:1035] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e8da74e740d4510a457fb5e6dd1d819" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:01:56.055502 15346 consensus_queue.cc:1035] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a300244bb80847f68991901ff6ed4f00" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:01:56.066947 15358 mvcc.cc:205] Tried to move back new op lower bound from 7317489729741520896 to 7317489729536585728. Current Snapshot: MvccSnapshot[applied={T|T < 7317489729741520896}]
I20260812 01:01:56.068022 15359 mvcc.cc:205] Tried to move back new op lower bound from 7317489729741520896 to 7317489729536585728. Current Snapshot: MvccSnapshot[applied={T|T < 7317489729741520896}]
I20260812 01:01:56.082778 14489 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:01:56.083164 14489 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:01:56.088842 14489 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790: Sending DeleteTablet for 3 replicas of tablet 642eb11d79104b79b5da8998c9fd48d5
I20260812 01:01:56.090601 14732 tablet_service.cc:1514] Processing DeleteTablet for tablet 642eb11d79104b79b5da8998c9fd48d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:01:56 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:41344
I20260812 01:01:56.090615 15189 tablet_service.cc:1514] Processing DeleteTablet for tablet 642eb11d79104b79b5da8998c9fd48d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:01:56 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:55966
I20260812 01:01:56.092044 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 14610
I20260812 01:01:56.093046 14973 tablet_service.cc:1514] Processing DeleteTablet for tablet 642eb11d79104b79b5da8998c9fd48d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:01:56 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:52792
I20260812 01:01:56.105876 15369 tablet_replica.cc:330] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3: stopping tablet replica
I20260812 01:01:56.106513 15369 raft_consensus.cc:2229] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3 [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:01:56.107386 15369 raft_consensus.cc:2258] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:01:56.108350 15374 tablet_replica.cc:330] T 642eb11d79104b79b5da8998c9fd48d5 P a300244bb80847f68991901ff6ed4f00: stopping tablet replica
I20260812 01:01:56.109046 15374 raft_consensus.cc:2229] T 642eb11d79104b79b5da8998c9fd48d5 P a300244bb80847f68991901ff6ed4f00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:01:56.109720 15374 raft_consensus.cc:2258] T 642eb11d79104b79b5da8998c9fd48d5 P a300244bb80847f68991901ff6ed4f00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:01:56.112303 15374 ts_tablet_manager.cc:1905] T 642eb11d79104b79b5da8998c9fd48d5 P a300244bb80847f68991901ff6ed4f00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:01:56.114414 15369 ts_tablet_manager.cc:1905] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:01:56.119550 15376 tablet_replica.cc:330] T 642eb11d79104b79b5da8998c9fd48d5 P 7e8da74e740d4510a457fb5e6dd1d819: stopping tablet replica
I20260812 01:01:56.120091 15376 raft_consensus.cc:2229] T 642eb11d79104b79b5da8998c9fd48d5 P 7e8da74e740d4510a457fb5e6dd1d819 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:01:56.120703 15376 raft_consensus.cc:2258] T 642eb11d79104b79b5da8998c9fd48d5 P 7e8da74e740d4510a457fb5e6dd1d819 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:01:56.124832 15374 ts_tablet_manager.cc:1918] T 642eb11d79104b79b5da8998c9fd48d5 P a300244bb80847f68991901ff6ed4f00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:01:56.125237 15374 log.cc:1196] T 642eb11d79104b79b5da8998c9fd48d5 P a300244bb80847f68991901ff6ed4f00: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/wal/wals/642eb11d79104b79b5da8998c9fd48d5
I20260812 01:01:56.126055 15374 ts_tablet_manager.cc:1939] T 642eb11d79104b79b5da8998c9fd48d5 P a300244bb80847f68991901ff6ed4f00: Deleting consensus metadata
I20260812 01:01:56.127156 15376 ts_tablet_manager.cc:1905] T 642eb11d79104b79b5da8998c9fd48d5 P 7e8da74e740d4510a457fb5e6dd1d819: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:01:56.128865 14471 catalog_manager.cc:4841] TS a300244bb80847f68991901ff6ed4f00 (127.21.201.65:35855): tablet 642eb11d79104b79b5da8998c9fd48d5 (table test-table [id=4de23b9c6f574c1bacf7ee6cf5000159]) successfully deleted
I20260812 01:01:56.130617 15369 ts_tablet_manager.cc:1918] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:01:56.130824 15369 log.cc:1196] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/642eb11d79104b79b5da8998c9fd48d5
I20260812 01:01:56.131492 15369 ts_tablet_manager.cc:1939] T 642eb11d79104b79b5da8998c9fd48d5 P 430435c9b2314d60b15673a369a0c5a3: Deleting consensus metadata
I20260812 01:01:56.133668 14469 catalog_manager.cc:4841] TS 430435c9b2314d60b15673a369a0c5a3 (127.21.201.67:34275): tablet 642eb11d79104b79b5da8998c9fd48d5 (table test-table [id=4de23b9c6f574c1bacf7ee6cf5000159]) successfully deleted
Aug 12 01:01:56 dist-test-slave-7f01 krb5kdc[14409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.65@KRBTEST.COM
I20260812 01:01:56.134853 15376 ts_tablet_manager.cc:1918] T 642eb11d79104b79b5da8998c9fd48d5 P 7e8da74e740d4510a457fb5e6dd1d819: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:01:56.135039 15376 log.cc:1196] T 642eb11d79104b79b5da8998c9fd48d5 P 7e8da74e740d4510a457fb5e6dd1d819: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/642eb11d79104b79b5da8998c9fd48d5
I20260812 01:01:56.135532 15376 ts_tablet_manager.cc:1939] T 642eb11d79104b79b5da8998c9fd48d5 P 7e8da74e740d4510a457fb5e6dd1d819: Deleting consensus metadata
I20260812 01:01:56.137727 14469 catalog_manager.cc:4841] TS 7e8da74e740d4510a457fb5e6dd1d819 (127.21.201.66:34007): tablet 642eb11d79104b79b5da8998c9fd48d5 (table test-table [id=4de23b9c6f574c1bacf7ee6cf5000159]) successfully deleted
I20260812 01:01:56.143641 15329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:56.094793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41356 (local address 127.21.201.65:35855)
0812 01:01:56.095055 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:56.095064 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:56.116268 (+ 21204us) server_negotiation.cc:407] Connection header received
0812 01:01:56.116619 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:56.116625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:56.116725 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:56.116809 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:56.121177 (+ 4368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:56.122622 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:56.127254 (+ 4632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:56.128006 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:56.128124 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:56.136168 (+ 8044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:56.136192 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:56.136214 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:56.136269 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:56.140012 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:56.141092 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:56.141104 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:56.141107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:56.141310 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:56.141822 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:56.141826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:56.141828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:56.142219 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:56.142478 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:56.142959 (+ 481us) server_negotiation.cc:299] Negotiation successful
0812 01:01:56.143236 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 01:01:56.144520 14840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:01:56.246745 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 14610
I20260812 01:01:56.280750 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 14856
Aug 12 01:01:56 dist-test-slave-7f01 krb5kdc[14409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.66@KRBTEST.COM
I20260812 01:01:56.299495 15333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:56.281714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52794 (local address 127.21.201.66:34007)
0812 01:01:56.281913 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:56.281917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:56.282037 (+ 120us) server_negotiation.cc:407] Connection header received
0812 01:01:56.282228 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:56.282232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:56.282305 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:56.282400 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:56.284181 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:56.284965 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:56.287078 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:56.287506 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:56.287613 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:56.293253 (+ 5640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:56.293271 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:56.293287 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:56.293324 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:56.296501 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:56.297245 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:56.297257 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:56.297260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:56.297398 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:56.297794 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:56.297801 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:56.297804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:56.298224 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:56.298551 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:56.298869 (+ 318us) server_negotiation.cc:299] Negotiation successful
0812 01:01:56.299113 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 01:01:56.300225 15059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:01:56.397806 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 14856
I20260812 01:01:56.430805 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 15082
Aug 12 01:01:56 dist-test-slave-7f01 krb5kdc[14409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.67@KRBTEST.COM
I20260812 01:01:56.454911 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:56.432147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55978 (local address 127.21.201.67:34275)
0812 01:01:56.432423 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:56.432429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:56.432525 (+ 96us) server_negotiation.cc:407] Connection header received
0812 01:01:56.432915 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:56.432922 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:56.433024 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:56.433172 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:56.435200 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:56.436555 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:56.438155 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:56.438906 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:56.439056 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:56.445125 (+ 6069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:56.445141 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:56.445159 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:56.445223 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:56.449777 (+ 4554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:56.451214 (+ 1437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:56.451239 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:56.451242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:56.451890 (+ 648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:56.452528 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:56.452535 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:56.452540 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:56.453057 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:56.453352 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:56.453832 (+ 480us) server_negotiation.cc:299] Negotiation successful
0812 01:01:56.454229 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260812 01:01:56.456136 15293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:01:56.594230 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 15082
I20260812 01:01:56.635355 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 14429
I20260812 01:01:56.637111 14541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:01:56.763270 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 14429
I20260812 01:01:56.804802 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:46059
--webserver_interface=127.21.201.126
--webserver_port=41783
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:46059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:01:57.184840 15388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:01:57.185168 15388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:01:57.185266 15388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:01:57.192675 15388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:01:57.192803 15388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:01:57.192852 15388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:01:57.192904 15388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:01:57.192952 15388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:01:57.203019 15388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:46059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:46059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=41783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:01:57.204813 15388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:01:57.206112 15388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:01:57.237169 15395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:57.237049 15399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:57.237138 15396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:01:57.238471 15388 server_base.cc:1034] running on GCE node
I20260812 01:01:57.239957 15388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:01:57.242298 15388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:01:57.243570 15388 hybrid_clock.cc:648] HybridClock initialized: now 1786496517243513 us; error 56 us; skew 500 ppm
I20260812 01:01:57.244118 15388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:01:57 dist-test-slave-7f01 krb5kdc[14409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496517, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:01:57.253739 15388 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 01:01:57.256062 15388 webserver.cc:460] Webserver started at http://127.21.201.126:41783/ using document root <none> and password file <none>
I20260812 01:01:57.256973 15388 fs_manager.cc:359] Metadata directory not provided
I20260812 01:01:57.257169 15388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:01:57.264735 15388 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 01:01:57.268620 15406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:57.270745 15388 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260812 01:01:57.270982 15388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "86b526280972413ebeff882ad481b790"
format_stamp: "Formatted at 2026-08-12 01:01:53 on dist-test-slave-7f01"
server_key: "af406f1ffed7fbf9f7baf5a90da6c526"
server_key_iv: "65375d5c4608cf8ef067171a16a0f6aa"
server_key_version: "encryptionkey@0"
I20260812 01:01:57.272280 15388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:01:57.301447 15388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:01:57.306512 15388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:01:57.306867 15388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:01:57.331512 15388 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:46059
I20260812 01:01:57.331533 15493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:46059 every 8 connection(s)
I20260812 01:01:57.333252 15388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:01:57.341390 15494 sys_catalog.cc:263] Verifying existing consensus state
I20260812 01:01:57.342594 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 15388
I20260812 01:01:57.343470 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:35855
--local_ip_for_outbound_sockets=127.21.201.65
--tserver_master_addrs=127.21.201.126:46059
--webserver_port=34645
--webserver_interface=127.21.201.65
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:01:57.345283 15494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790: Bootstrap starting.
I20260812 01:01:57.371171 15494 log.cc:824] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790: Log is configured to *not* fsync() on all Append() calls
I20260812 01:01:57.389454 15494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 01:01:57.390249 15494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790: Bootstrap complete.
I20260812 01:01:57.404673 15494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b526280972413ebeff882ad481b790" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46059 } }
I20260812 01:01:57.406003 15494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b526280972413ebeff882ad481b790, State: Initialized, Role: FOLLOWER
I20260812 01:01:57.406884 15494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b526280972413ebeff882ad481b790" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46059 } }
I20260812 01:01:57.407208 15494 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:01:57.407347 15494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:01:57.407557 15494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [term 1 FOLLOWER]: Advancing to term 2
I20260812 01:01:57.413751 15494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b526280972413ebeff882ad481b790" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46059 } }
I20260812 01:01:57.414541 15494 leader_election.cc:302] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86b526280972413ebeff882ad481b790; no voters:
I20260812 01:01:57.415320 15494 leader_election.cc:288] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 01:01:57.415529 15499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 01:01:57.417011 15499 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [term 2 LEADER]: Becoming Leader. State: Replica: 86b526280972413ebeff882ad481b790, State: Running, Role: LEADER
I20260812 01:01:57.417616 15499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b526280972413ebeff882ad481b790" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46059 } }
I20260812 01:01:57.418020 15494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:01:57.420548 15503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86b526280972413ebeff882ad481b790. Latest consensus state: current_term: 2 leader_uuid: "86b526280972413ebeff882ad481b790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b526280972413ebeff882ad481b790" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46059 } } }
I20260812 01:01:57.420495 15502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "86b526280972413ebeff882ad481b790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b526280972413ebeff882ad481b790" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46059 } } }
I20260812 01:01:57.420941 15502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [sys.catalog]: This master's current role is: LEADER
I20260812 01:01:57.420941 15503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [sys.catalog]: This master's current role is: LEADER
I20260812 01:01:57.436882 15510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:01:57.453698 15510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:01:57.457557 15510 catalog_manager.cc:1259] Loaded cluster ID: 9f307e7a184d412fa2314b73e672dead
I20260812 01:01:57.458092 15510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:01:57.466123 15510 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260812 01:01:57.503437 15510 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260812 01:01:57.521359 15510 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:01:57.523677 15510 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790: Loaded TSK: 0
I20260812 01:01:57.526377 15510 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 01:01:57.836524 15497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:01:57.836899 15497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:01:57.837085 15497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:01:57.846863 15497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:01:57.847054 15497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:01:57.847247 15497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 01:01:57.858088 15497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:35855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=34645
--tserver_master_addrs=127.21.201.126:46059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:01:57.859999 15497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:01:57.861485 15497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:01:57.882104 15552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:57.885373 15551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:57.885860 15554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:01:57.887501 15497 server_base.cc:1034] running on GCE node
I20260812 01:01:57.888509 15497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:01:57.890156 15497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:01:57.891440 15497 hybrid_clock.cc:648] HybridClock initialized: now 1786496517891374 us; error 47 us; skew 500 ppm
I20260812 01:01:57.891878 15497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:01:57 dist-test-slave-7f01 krb5kdc[14409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496517, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:01:57.896703 15497 init.cc:377] Logged in from keytab as kudu/127.21.201.65@KRBTEST.COM (short username kudu)
I20260812 01:01:57.899015 15497 webserver.cc:460] Webserver started at http://127.21.201.65:34645/ using document root <none> and password file <none>
I20260812 01:01:57.900120 15497 fs_manager.cc:359] Metadata directory not provided
I20260812 01:01:57.900322 15497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:01:57.907286 15497 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 01:01:57.910645 15568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:57.912951 15497 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 01:01:57.913192 15497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "a300244bb80847f68991901ff6ed4f00"
format_stamp: "Formatted at 2026-08-12 01:01:54 on dist-test-slave-7f01"
server_key: "dd08d3277f3e90779b064b315e917363"
server_key_iv: "1d4825eee7ddf41a0f3d17287fdd13b4"
server_key_version: "encryptionkey@0"
I20260812 01:01:57.914594 15497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:01:57.927557 15497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:01:57.932385 15497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:01:57.932690 15497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:01:57.933796 15497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:01:57.935899 15497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:01:57.936110 15497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 01:01:57.936422 15497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:01:57.936527 15497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:57.980626 15497 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:35855
I20260812 01:01:57.980738 15758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:35855 every 8 connection(s)
I20260812 01:01:57.982127 15497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 01:01:57.988144 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 15497
I20260812 01:01:57.988848 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:34007
--local_ip_for_outbound_sockets=127.21.201.66
--tserver_master_addrs=127.21.201.126:46059
--webserver_port=36363
--webserver_interface=127.21.201.66
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 01:01:57 dist-test-slave-7f01 krb5kdc[14409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496517, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:01:58.012841 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:57.986081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:52557 (local address 127.21.201.126:46059)
0812 01:01:57.986792 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:57.986804 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:57.987806 (+ 1002us) server_negotiation.cc:407] Connection header received
0812 01:01:57.990888 (+ 3082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:57.990921 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:57.991643 (+ 722us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:57.992291 (+ 648us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:57.994450 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:57.996858 (+ 2408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:57.998586 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:57.999689 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:57.999923 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:58.004168 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:58.004189 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:58.004215 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:58.004280 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:58.008053 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:58.008901 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:58.008909 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:58.008915 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:58.009086 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:58.009529 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:58.009535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:58.009537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:58.010034 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:58.010241 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:58.011045 (+ 804us) server_negotiation.cc:299] Negotiation successful
0812 01:01:58.011478 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":302,"threads_started":1}
I20260812 01:01:58.012878 15764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:57.986367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.65:52557)
0812 01:01:57.987525 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:57.987608 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:57.989848 (+ 2240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:57.992640 (+ 2792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:57.992653 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:57.993471 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:01:57.994154 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:57.994174 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:57.997113 (+ 2939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:57.997118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:57.998394 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:57.998419 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:57.999978 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:57.999984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:58.000188 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:58.000979 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:01:58.001004 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:01:58.003599 (+ 2595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:01:58.008239 (+ 4640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:58.008247 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:58.008263 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:58.008740 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:58.009224 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:58.009228 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:58.009230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:58.009410 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:58.010191 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:01:58.010199 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:01:58.010527 (+ 328us) client_negotiation.cc:769] Sending connection context
0812 01:01:58.010809 (+ 282us) client_negotiation.cc:240] Negotiation successful
0812 01:01:58.011460 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":432,"threads_started":1}
I20260812 01:01:58.016811 15759 heartbeater.cc:347] Connected to a master server at 127.21.201.126:46059
I20260812 01:01:58.017174 15759 heartbeater.cc:464] Registering TS with master...
I20260812 01:01:58.017978 15759 heartbeater.cc:511] Master 127.21.201.126:46059 requested a full tablet report, sending...
I20260812 01:01:58.020751 15432 ts_manager.cc:194] Registered new tserver with Master: a300244bb80847f68991901ff6ed4f00 (127.21.201.65:35855)
I20260812 01:01:58.022238 15432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.65@KRBTEST.COM'} at 127.21.201.65:52557
W20260812 01:01:58.357504 15766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:01:58.357795 15766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:01:58.357996 15766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:01:58.365577 15766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:01:58.365701 15766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:01:58.365927 15766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 01:01:58.376420 15766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:34007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=36363
--tserver_master_addrs=127.21.201.126:46059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:01:58.377897 15766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:01:58.379254 15766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:01:58.392699 15777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:58.392643 15781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:58.392740 15776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:01:58.393714 15766 server_base.cc:1034] running on GCE node
I20260812 01:01:58.394435 15766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:01:58.395401 15766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:01:58.396620 15766 hybrid_clock.cc:648] HybridClock initialized: now 1786496518396576 us; error 33 us; skew 500 ppm
I20260812 01:01:58.396970 15766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:01:58 dist-test-slave-7f01 krb5kdc[14409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496518, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:01:58.400238 15766 init.cc:377] Logged in from keytab as kudu/127.21.201.66@KRBTEST.COM (short username kudu)
I20260812 01:01:58.401723 15766 webserver.cc:460] Webserver started at http://127.21.201.66:36363/ using document root <none> and password file <none>
I20260812 01:01:58.402426 15766 fs_manager.cc:359] Metadata directory not provided
I20260812 01:01:58.402602 15766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:01:58.408044 15766 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 01:01:58.410725 15792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:58.412096 15766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:01:58.412274 15766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "7e8da74e740d4510a457fb5e6dd1d819"
format_stamp: "Formatted at 2026-08-12 01:01:55 on dist-test-slave-7f01"
server_key: "1f2696dfc83d5691f9e1af5b3585ba38"
server_key_iv: "f8e645c442be961a09a5b5c1b6bf1b32"
server_key_version: "encryptionkey@0"
I20260812 01:01:58.413282 15766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:01:58.442965 15766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:01:58.447687 15766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:01:58.448045 15766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:01:58.449100 15766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:01:58.450670 15766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:01:58.450796 15766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:58.450938 15766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:01:58.451037 15766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:58.486068 15766 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:34007
I20260812 01:01:58.486166 15988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:34007 every 8 connection(s)
I20260812 01:01:58.487613 15766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 01:01:58.497848 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 15766
I20260812 01:01:58.498533 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:34275
--local_ip_for_outbound_sockets=127.21.201.67
--tserver_master_addrs=127.21.201.126:46059
--webserver_port=38743
--webserver_interface=127.21.201.67
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 01:01:58 dist-test-slave-7f01 krb5kdc[14409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496518, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:01:58.508504 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:58.490334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:47729 (local address 127.21.201.126:46059)
0812 01:01:58.490530 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:58.490535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:58.491691 (+ 1156us) server_negotiation.cc:407] Connection header received
0812 01:01:58.493628 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:58.493634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:58.493760 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:58.493880 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:58.495393 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:58.496207 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:58.497893 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:58.498370 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:58.498511 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:58.502388 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:58.502394 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:58.502403 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:58.502443 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:58.505254 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:58.506032 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:58.506040 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:58.506043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:58.506188 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:58.506649 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:58.506655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:58.506658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:58.506956 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:58.507132 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:58.507857 (+ 725us) server_negotiation.cc:299] Negotiation successful
0812 01:01:58.508099 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 01:01:58.509519 15993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:58.490808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.66:47729)
0812 01:01:58.491532 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:58.491575 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:58.493360 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:58.494123 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:58.494131 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:58.494647 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:01:58.495220 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:58.495236 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:58.496361 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:58.496371 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:58.497742 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:58.497754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:58.498604 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:58.498612 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:58.498875 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:58.499611 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:01:58.499682 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:01:58.502158 (+ 2476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:01:58.505404 (+ 3246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:58.505415 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:58.505431 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:58.505879 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:58.506308 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:58.506312 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:58.506316 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:58.506525 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:58.507110 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:01:58.507118 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:01:58.507461 (+ 343us) client_negotiation.cc:769] Sending connection context
0812 01:01:58.507731 (+ 270us) client_negotiation.cc:240] Negotiation successful
0812 01:01:58.508324 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":286,"threads_started":1}
I20260812 01:01:58.511157 15989 heartbeater.cc:347] Connected to a master server at 127.21.201.126:46059
I20260812 01:01:58.511493 15989 heartbeater.cc:464] Registering TS with master...
I20260812 01:01:58.512563 15989 heartbeater.cc:511] Master 127.21.201.126:46059 requested a full tablet report, sending...
I20260812 01:01:58.514375 15432 ts_manager.cc:194] Registered new tserver with Master: 7e8da74e740d4510a457fb5e6dd1d819 (127.21.201.66:34007)
I20260812 01:01:58.515329 15432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.66@KRBTEST.COM'} at 127.21.201.66:47729
W20260812 01:01:58.874925 15995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:01:58.875218 15995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:01:58.875403 15995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:01:58.883304 15995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:01:58.883430 15995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:01:58.883663 15995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 01:01:58.893934 15995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:34275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=38743
--tserver_master_addrs=127.21.201.126:46059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:01:58.895620 15995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:01:58.897480 15995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:01:58.912962 16006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:58.912995 16004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:01:58.913034 16002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:01:58.913992 15995 server_base.cc:1034] running on GCE node
I20260812 01:01:58.914769 15995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:01:58.915858 15995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:01:58.917102 15995 hybrid_clock.cc:648] HybridClock initialized: now 1786496518917041 us; error 46 us; skew 500 ppm
I20260812 01:01:58.917521 15995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:01:58 dist-test-slave-7f01 krb5kdc[14409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496518, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:01:58.921167 15995 init.cc:377] Logged in from keytab as kudu/127.21.201.67@KRBTEST.COM (short username kudu)
I20260812 01:01:58.922857 15995 webserver.cc:460] Webserver started at http://127.21.201.67:38743/ using document root <none> and password file <none>
I20260812 01:01:58.923597 15995 fs_manager.cc:359] Metadata directory not provided
I20260812 01:01:58.923771 15995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:01:58.929322 15995 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260812 01:01:58.932808 16020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:58.934232 15995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 01:01:58.934441 15995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "430435c9b2314d60b15673a369a0c5a3"
format_stamp: "Formatted at 2026-08-12 01:01:55 on dist-test-slave-7f01"
server_key: "a328c36199471f699ec4562866f8f5aa"
server_key_iv: "16e8525042db1c138c59b8a98926fc4e"
server_key_version: "encryptionkey@0"
I20260812 01:01:58.935460 15995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:01:58.953507 15995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:01:58.957845 15995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:01:58.958161 15995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:01:58.959268 15995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:01:58.960752 15995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:01:58.960850 15995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:58.960992 15995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:01:58.961118 15995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:01:58.997295 15995 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:34275
I20260812 01:01:58.997411 16207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:34275 every 8 connection(s)
I20260812 01:01:58.998862 15995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 01:01:59.003049 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 15995
Aug 12 01:01:59 dist-test-slave-7f01 krb5kdc[14409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496518, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:01:59.021016 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:59.004067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43500 (local address 127.21.201.126:46059)
0812 01:01:59.004570 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:59.004577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:59.004904 (+ 327us) server_negotiation.cc:407] Connection header received
0812 01:01:59.005056 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:59.005062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:59.005149 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:59.005283 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:59.007009 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.007996 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.009413 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.010140 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.010231 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:59.013684 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:59.013690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:59.013693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:59.013725 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:59.017286 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:59.018396 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:59.018402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:59.018406 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:59.018586 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:59.019197 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:59.019205 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:59.019211 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:59.019602 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:59.019892 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:59.020295 (+ 403us) server_negotiation.cc:299] Negotiation successful
0812 01:01:59.020544 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":244,"threads_started":1}
I20260812 01:01:59.022478 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:59.001637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:41197 (local address 127.21.201.126:46059)
0812 01:01:59.001872 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:59.001879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:59.003013 (+ 1134us) server_negotiation.cc:407] Connection header received
0812 01:01:59.005447 (+ 2434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:59.005459 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:59.005545 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:59.005647 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:59.007208 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.008156 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.010369 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.011069 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.011215 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:59.015134 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:59.015142 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:59.015146 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:59.015187 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:59.018546 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:59.019732 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:59.019738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:59.019742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:59.019960 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:59.020511 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:59.020518 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:59.020521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:59.020852 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:59.020961 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:59.021882 (+ 921us) server_negotiation.cc:299] Negotiation successful
0812 01:01:59.022116 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":84}
I20260812 01:01:59.023921 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:59.002102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.67:41197)
0812 01:01:59.002843 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:59.002881 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:59.005133 (+ 2252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:59.005858 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:59.005865 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:59.006384 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:01:59.007013 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:59.007042 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.008377 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.008381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:59.009842 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:59.009858 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.011351 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.011358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:59.011591 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:59.012309 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:01:59.012331 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:01:59.014896 (+ 2565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:01:59.018716 (+ 3820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:59.018728 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:59.018765 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:59.019579 (+ 814us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:59.020080 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:01:59.020084 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:01:59.020086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:01:59.020366 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:01:59.021002 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:01:59.021012 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:01:59.021416 (+ 404us) client_negotiation.cc:769] Sending connection context
0812 01:01:59.021765 (+ 349us) client_negotiation.cc:240] Negotiation successful
0812 01:01:59.022399 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":329,"threads_started":1}
I20260812 01:01:59.025406 16208 heartbeater.cc:347] Connected to a master server at 127.21.201.126:46059
I20260812 01:01:59.025645 15759 heartbeater.cc:503] Master 127.21.201.126:46059 was elected leader, sending a full tablet report...
I20260812 01:01:59.025743 16208 heartbeater.cc:464] Registering TS with master...
I20260812 01:01:59.026582 16208 heartbeater.cc:511] Master 127.21.201.126:46059 requested a full tablet report, sending...
I20260812 01:01:59.028113 15432 ts_manager.cc:194] Registered new tserver with Master: 430435c9b2314d60b15673a369a0c5a3 (127.21.201.67:34275)
I20260812 01:01:59.028869 15432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.67@KRBTEST.COM'} at 127.21.201.67:41197
I20260812 01:01:59.036409 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:01:59.044209 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:59.037315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43508 (local address 127.21.201.126:46059)
0812 01:01:59.037582 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:59.037589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:59.037895 (+ 306us) server_negotiation.cc:407] Connection header received
0812 01:01:59.038079 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:59.038083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:59.038207 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:59.038326 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:01:59.038741 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.039503 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.042009 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.042478 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.042582 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:59.043101 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:01:59.043577 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:01:59.043693 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:59.043835 (+ 142us) server_negotiation.cc:299] Negotiation successful
0812 01:01:59.043911 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260812 01:01:59.045117 15432 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:01:59.052933 15432 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:01:59.088109 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:59.068642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49754 (local address 127.21.201.66:34007)
0812 01:01:59.079586 (+ 10944us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:59.079598 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:59.079632 (+ 34us) server_negotiation.cc:407] Connection header received
0812 01:01:59.079776 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:59.079785 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:59.080097 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:59.081413 (+ 1316us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:01:59.081697 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.082770 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.085427 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.086826 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.086992 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:59.087224 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:59.087521 (+ 297us) server_negotiation.cc:299] Negotiation successful
0812 01:01:59.087699 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10738,"thread_start_us":10581,"threads_started":1}
I20260812 01:01:59.088186 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:59.068388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:34007 (local address 127.0.0.1:49754)
0812 01:01:59.079294 (+ 10906us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:59.079314 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:59.079448 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:59.080557 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:59.080565 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:59.080608 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:01:59.081191 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:59.081204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.082928 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.082934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:59.085224 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:59.085238 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.087053 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.087060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:59.087242 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:59.087315 (+ 73us) client_negotiation.cc:769] Sending connection context
0812 01:01:59.087465 (+ 150us) client_negotiation.cc:240] Negotiation successful
0812 01:01:59.087658 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10739,"mutex_wait_us":182,"spinlock_wait_cycles":896,"thread_start_us":11103,"threads_started":1}
I20260812 01:01:59.089072 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:59.065969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:35855 (local address 127.0.0.1:47396)
0812 01:01:59.067101 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:59.067185 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:59.067549 (+ 364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:59.080556 (+ 13007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:59.080565 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:59.080614 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:01:59.081430 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:59.081442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.083497 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.083506 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:59.086150 (+ 2644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:59.086166 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.088143 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.088150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:59.088348 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:59.088392 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 01:01:59.088462 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 01:01:59.088588 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":362,"spinlock_wait_cycles":9984,"thread_start_us":564,"threads_started":1}
I20260812 01:01:59.089226 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:59.066395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:34275 (local address 127.0.0.1:38448)
0812 01:01:59.068247 (+ 1852us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:59.068267 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:59.074696 (+ 6429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:59.080958 (+ 6262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:59.080963 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:59.081011 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:01:59.081633 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:59.081649 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.083626 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.083633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:59.086151 (+ 2518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:59.086165 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.088416 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.088423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:59.088579 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:59.088607 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 01:01:59.088688 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 01:01:59.088818 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1712,"mutex_wait_us":55,"spinlock_wait_cycles":2816,"thread_start_us":1580,"threads_started":1}
I20260812 01:01:59.089547 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:59.066733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47396 (local address 127.21.201.65:35855)
0812 01:01:59.079833 (+ 13100us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:59.079841 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:59.079866 (+ 25us) server_negotiation.cc:407] Connection header received
0812 01:01:59.079952 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:59.079958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:59.080137 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:59.080702 (+ 565us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:01:59.081633 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.083308 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.086339 (+ 3031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.087884 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.088731 (+ 847us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:59.088962 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:59.089030 (+ 68us) server_negotiation.cc:299] Negotiation successful
0812 01:01:59.089183 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12919,"thread_start_us":13502,"threads_started":1}
I20260812 01:01:59.090071 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:59.067533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38448 (local address 127.21.201.67:34275)
0812 01:01:59.080057 (+ 12524us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:59.080063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:59.080090 (+ 27us) server_negotiation.cc:407] Connection header received
0812 01:01:59.080227 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:59.080234 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:59.080464 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:59.080675 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:01:59.081834 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.083448 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.086347 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.087877 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.088017 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:59.088251 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:59.089499 (+ 1248us) server_negotiation.cc:299] Negotiation successful
0812 01:01:59.089669 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12351,"thread_start_us":12678,"threads_started":1}
I20260812 01:01:59.093209 16104 tablet_service.cc:1467] Processing CreateTablet for tablet c0da7326ddff4b029a6a1ea09f690052 (DEFAULT_TABLE table=test-table [id=7218cc6a461c47ddb3e4212fd2c7780c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:01:59.093140 15651 tablet_service.cc:1467] Processing CreateTablet for tablet c0da7326ddff4b029a6a1ea09f690052 (DEFAULT_TABLE table=test-table [id=7218cc6a461c47ddb3e4212fd2c7780c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:01:59.093142 15873 tablet_service.cc:1467] Processing CreateTablet for tablet c0da7326ddff4b029a6a1ea09f690052 (DEFAULT_TABLE table=test-table [id=7218cc6a461c47ddb3e4212fd2c7780c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:01:59.094462 16104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0da7326ddff4b029a6a1ea09f690052. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:01:59.094525 15873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0da7326ddff4b029a6a1ea09f690052. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:01:59.094542 15651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0da7326ddff4b029a6a1ea09f690052. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:01:59.109372 16231 tablet_bootstrap.cc:492] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3: Bootstrap starting.
I20260812 01:01:59.111619 16233 tablet_bootstrap.cc:492] T c0da7326ddff4b029a6a1ea09f690052 P a300244bb80847f68991901ff6ed4f00: Bootstrap starting.
I20260812 01:01:59.112427 16234 tablet_bootstrap.cc:492] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819: Bootstrap starting.
I20260812 01:01:59.113242 16231 tablet_bootstrap.cc:654] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3: Neither blocks nor log segments found. Creating new log.
I20260812 01:01:59.114626 16231 log.cc:824] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3: Log is configured to *not* fsync() on all Append() calls
I20260812 01:01:59.115532 16233 tablet_bootstrap.cc:654] T c0da7326ddff4b029a6a1ea09f690052 P a300244bb80847f68991901ff6ed4f00: Neither blocks nor log segments found. Creating new log.
I20260812 01:01:59.115978 16234 tablet_bootstrap.cc:654] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819: Neither blocks nor log segments found. Creating new log.
I20260812 01:01:59.116667 16233 log.cc:824] T c0da7326ddff4b029a6a1ea09f690052 P a300244bb80847f68991901ff6ed4f00: Log is configured to *not* fsync() on all Append() calls
I20260812 01:01:59.117288 16234 log.cc:824] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819: Log is configured to *not* fsync() on all Append() calls
I20260812 01:01:59.119132 16231 tablet_bootstrap.cc:492] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3: No bootstrap required, opened a new log
I20260812 01:01:59.119529 16231 ts_tablet_manager.cc:1397] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260812 01:01:59.119895 16233 tablet_bootstrap.cc:492] T c0da7326ddff4b029a6a1ea09f690052 P a300244bb80847f68991901ff6ed4f00: No bootstrap required, opened a new log
I20260812 01:01:59.120257 16233 ts_tablet_manager.cc:1397] T c0da7326ddff4b029a6a1ea09f690052 P a300244bb80847f68991901ff6ed4f00: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260812 01:01:59.122269 16234 tablet_bootstrap.cc:492] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819: No bootstrap required, opened a new log
I20260812 01:01:59.122527 16234 ts_tablet_manager.cc:1397] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260812 01:01:59.128072 16233 raft_consensus.cc:348] T c0da7326ddff4b029a6a1ea09f690052 P a300244bb80847f68991901ff6ed4f00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a300244bb80847f68991901ff6ed4f00" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35855 } } peers { permanent_uuid: "430435c9b2314d60b15673a369a0c5a3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34275 } } peers { permanent_uuid: "7e8da74e740d4510a457fb5e6dd1d819" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34007 } }
I20260812 01:01:59.128453 16233 raft_consensus.cc:374] T c0da7326ddff4b029a6a1ea09f690052 P a300244bb80847f68991901ff6ed4f00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:01:59.128594 16233 raft_consensus.cc:729] T c0da7326ddff4b029a6a1ea09f690052 P a300244bb80847f68991901ff6ed4f00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a300244bb80847f68991901ff6ed4f00, State: Initialized, Role: FOLLOWER
I20260812 01:01:59.129199 16233 consensus_queue.cc:260] T c0da7326ddff4b029a6a1ea09f690052 P a300244bb80847f68991901ff6ed4f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a300244bb80847f68991901ff6ed4f00" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35855 } } peers { permanent_uuid: "430435c9b2314d60b15673a369a0c5a3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34275 } } peers { permanent_uuid: "7e8da74e740d4510a457fb5e6dd1d819" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34007 } }
I20260812 01:01:59.130448 16231 raft_consensus.cc:348] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a300244bb80847f68991901ff6ed4f00" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35855 } } peers { permanent_uuid: "430435c9b2314d60b15673a369a0c5a3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34275 } } peers { permanent_uuid: "7e8da74e740d4510a457fb5e6dd1d819" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34007 } }
I20260812 01:01:59.130908 16231 raft_consensus.cc:374] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:01:59.130952 16233 ts_tablet_manager.cc:1428] T c0da7326ddff4b029a6a1ea09f690052 P a300244bb80847f68991901ff6ed4f00: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260812 01:01:59.131028 16231 raft_consensus.cc:729] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 430435c9b2314d60b15673a369a0c5a3, State: Initialized, Role: FOLLOWER
I20260812 01:01:59.131731 16231 consensus_queue.cc:260] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a300244bb80847f68991901ff6ed4f00" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35855 } } peers { permanent_uuid: "430435c9b2314d60b15673a369a0c5a3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34275 } } peers { permanent_uuid: "7e8da74e740d4510a457fb5e6dd1d819" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34007 } }
I20260812 01:01:59.132978 16208 heartbeater.cc:503] Master 127.21.201.126:46059 was elected leader, sending a full tablet report...
I20260812 01:01:59.133563 16231 ts_tablet_manager.cc:1428] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3: Time spent starting tablet: real 0.014s user 0.004s sys 0.009s
I20260812 01:01:59.133944 16234 raft_consensus.cc:348] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a300244bb80847f68991901ff6ed4f00" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35855 } } peers { permanent_uuid: "430435c9b2314d60b15673a369a0c5a3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34275 } } peers { permanent_uuid: "7e8da74e740d4510a457fb5e6dd1d819" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34007 } }
I20260812 01:01:59.134440 16234 raft_consensus.cc:374] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:01:59.134548 16234 raft_consensus.cc:729] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e8da74e740d4510a457fb5e6dd1d819, State: Initialized, Role: FOLLOWER
I20260812 01:01:59.135094 16234 consensus_queue.cc:260] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a300244bb80847f68991901ff6ed4f00" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35855 } } peers { permanent_uuid: "430435c9b2314d60b15673a369a0c5a3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34275 } } peers { permanent_uuid: "7e8da74e740d4510a457fb5e6dd1d819" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34007 } }
I20260812 01:01:59.136462 15989 heartbeater.cc:503] Master 127.21.201.126:46059 was elected leader, sending a full tablet report...
I20260812 01:01:59.136992 16234 ts_tablet_manager.cc:1428] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260812 01:01:59.235056 15760 tablet.cc:2369] T c0da7326ddff4b029a6a1ea09f690052 P a300244bb80847f68991901ff6ed4f00: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:01:59.239987 15992 tablet.cc:2369] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:01:59.250665 16209 tablet.cc:2369] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:01:59.572407 16241 raft_consensus.cc:482] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:01:59.572731 16241 raft_consensus.cc:504] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a300244bb80847f68991901ff6ed4f00" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35855 } } peers { permanent_uuid: "430435c9b2314d60b15673a369a0c5a3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34275 } } peers { permanent_uuid: "7e8da74e740d4510a457fb5e6dd1d819" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34007 } }
I20260812 01:01:59.574515 16241 leader_election.cc:288] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a300244bb80847f68991901ff6ed4f00 (127.21.201.65:35855), 7e8da74e740d4510a457fb5e6dd1d819 (127.21.201.66:34007)
I20260812 01:01:59.579573 16244 raft_consensus.cc:482] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:01:59.579929 16244 raft_consensus.cc:504] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a300244bb80847f68991901ff6ed4f00" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35855 } } peers { permanent_uuid: "430435c9b2314d60b15673a369a0c5a3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34275 } } peers { permanent_uuid: "7e8da74e740d4510a457fb5e6dd1d819" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34007 } }
I20260812 01:01:59.581979 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:59.575087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:46373 (local address 127.21.201.65:35855)
0812 01:01:59.575515 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:59.575522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:59.575914 (+ 392us) server_negotiation.cc:407] Connection header received
0812 01:01:59.576181 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:59.576190 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:59.576301 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:59.576397 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:01:59.577009 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.577907 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.580101 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.581057 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.581237 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:59.581315 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:59.581574 (+ 259us) server_negotiation.cc:299] Negotiation successful
0812 01:01:59.581700 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263}
I20260812 01:01:59.582034 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:59.574780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:35855 (local address 127.21.201.67:46373)
0812 01:01:59.575771 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:59.575796 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:59.575989 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:59.576437 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:59.576442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:59.576465 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:01:59.576836 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:59.576847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.578058 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.578063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:59.579922 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:59.579939 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.581220 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.581226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:59.581411 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:59.581437 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 01:01:59.581548 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 01:01:59.581689 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":980,"threads_started":1}
I20260812 01:01:59.582139 16244 leader_election.cc:288] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a300244bb80847f68991901ff6ed4f00 (127.21.201.65:35855), 430435c9b2314d60b15673a369a0c5a3 (127.21.201.67:34275)
I20260812 01:01:59.583359 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:59.575032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:51429 (local address 127.21.201.66:34007)
0812 01:01:59.575307 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:59.575316 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:59.576271 (+ 955us) server_negotiation.cc:407] Connection header received
0812 01:01:59.576415 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:59.576420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:59.576508 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:59.576618 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:01:59.577448 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.578378 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.580616 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.581863 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.582011 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:59.582089 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:59.582727 (+ 638us) server_negotiation.cc:299] Negotiation successful
0812 01:01:59.583065 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 01:01:59.583273 15685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0da7326ddff4b029a6a1ea09f690052" candidate_uuid: "430435c9b2314d60b15673a369a0c5a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a300244bb80847f68991901ff6ed4f00" is_pre_election: true
I20260812 01:01:59.583949 15685 raft_consensus.cc:2454] T c0da7326ddff4b029a6a1ea09f690052 P a300244bb80847f68991901ff6ed4f00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 430435c9b2314d60b15673a369a0c5a3 in term 0.
I20260812 01:01:59.584244 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:59.574940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:34007 (local address 127.21.201.67:51429)
0812 01:01:59.576134 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:59.576152 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:59.576277 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:59.576645 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:59.576649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:59.576669 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:01:59.577178 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:59.577191 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.578528 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.578531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:59.580435 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:59.580450 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.582295 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.582306 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:59.582498 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:59.582552 (+ 54us) client_negotiation.cc:769] Sending connection context
0812 01:01:59.583630 (+ 1078us) client_negotiation.cc:240] Negotiation successful
0812 01:01:59.583805 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1050,"thread_start_us":389,"threads_started":1}
I20260812 01:01:59.584794 16024 leader_election.cc:302] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 430435c9b2314d60b15673a369a0c5a3, a300244bb80847f68991901ff6ed4f00; no voters:
I20260812 01:01:59.585319 16241 raft_consensus.cc:2781] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:01:59.585475 16241 raft_consensus.cc:482] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:01:59.585614 16241 raft_consensus.cc:3037] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:01:59.585831 15912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0da7326ddff4b029a6a1ea09f690052" candidate_uuid: "430435c9b2314d60b15673a369a0c5a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e8da74e740d4510a457fb5e6dd1d819" is_pre_election: true
I20260812 01:01:59.586334 15912 raft_consensus.cc:2454] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 430435c9b2314d60b15673a369a0c5a3 in term 0.
I20260812 01:01:59.589243 16241 raft_consensus.cc:504] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a300244bb80847f68991901ff6ed4f00" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35855 } } peers { permanent_uuid: "430435c9b2314d60b15673a369a0c5a3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34275 } } peers { permanent_uuid: "7e8da74e740d4510a457fb5e6dd1d819" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34007 } }
I20260812 01:01:59.590492 15912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0da7326ddff4b029a6a1ea09f690052" candidate_uuid: "430435c9b2314d60b15673a369a0c5a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e8da74e740d4510a457fb5e6dd1d819"
I20260812 01:01:59.590718 15912 raft_consensus.cc:3037] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:01:59.592010 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:59.583329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:55745 (local address 127.21.201.65:35855)
0812 01:01:59.584447 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:59.584455 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:59.584474 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:01:59.584531 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:59.584537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:59.584606 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:59.584695 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:01:59.585519 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.586595 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.589349 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.590604 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.590761 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:59.590835 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:59.591542 (+ 707us) server_negotiation.cc:299] Negotiation successful
0812 01:01:59.591697 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":975}
I20260812 01:01:59.592084 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:59.582791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:34803 (local address 127.21.201.67:34275)
0812 01:01:59.582963 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:59.582970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:59.585014 (+ 2044us) server_negotiation.cc:407] Connection header received
0812 01:01:59.585307 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:59.585312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:59.585388 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:59.585453 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:01:59.586419 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.587547 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.589842 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.590971 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.591068 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:59.591114 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:59.591693 (+ 579us) server_negotiation.cc:299] Negotiation successful
0812 01:01:59.591850 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 01:01:59.592094 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:59.582929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:35855 (local address 127.21.201.66:55745)
0812 01:01:59.583700 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:59.583721 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:59.583874 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:59.584860 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:59.584866 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:59.584894 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:01:59.585321 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:59.585337 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.586796 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.586804 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:59.589141 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:59.589158 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.591072 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.591079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:59.591297 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:59.591388 (+ 91us) client_negotiation.cc:769] Sending connection context
0812 01:01:59.591499 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 01:01:59.591636 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":4,"thread_start_us":1175,"threads_started":1}
I20260812 01:01:59.592604 15685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0da7326ddff4b029a6a1ea09f690052" candidate_uuid: "430435c9b2314d60b15673a369a0c5a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a300244bb80847f68991901ff6ed4f00"
I20260812 01:01:59.592847 15685 raft_consensus.cc:3037] T c0da7326ddff4b029a6a1ea09f690052 P a300244bb80847f68991901ff6ed4f00 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:01:59.592905 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:59.583667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:34275 (local address 127.21.201.66:34803)
0812 01:01:59.584917 (+ 1250us) negotiation.cc:107] Waiting for socket to connect
0812 01:01:59.584933 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 01:01:59.585128 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:01:59.585644 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:01:59.585649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:01:59.585674 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:01:59.586174 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:59.586188 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.587707 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.587712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:59.589489 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:01:59.589507 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.591296 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.591301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:01:59.591513 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:59.591544 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 01:01:59.592302 (+ 758us) client_negotiation.cc:240] Negotiation successful
0812 01:01:59.592450 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1125,"mutex_wait_us":70,"thread_start_us":954,"threads_started":1}
I20260812 01:01:59.593755 16134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0da7326ddff4b029a6a1ea09f690052" candidate_uuid: "7e8da74e740d4510a457fb5e6dd1d819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "430435c9b2314d60b15673a369a0c5a3" is_pre_election: true
I20260812 01:01:59.594195 16134 raft_consensus.cc:2379] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e8da74e740d4510a457fb5e6dd1d819 in current term 1: Already voted for candidate 430435c9b2314d60b15673a369a0c5a3 in this term.
I20260812 01:01:59.594367 15912 raft_consensus.cc:2454] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 430435c9b2314d60b15673a369a0c5a3 in term 1.
I20260812 01:01:59.594870 16022 leader_election.cc:302] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 430435c9b2314d60b15673a369a0c5a3, 7e8da74e740d4510a457fb5e6dd1d819; no voters:
I20260812 01:01:59.594833 15683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0da7326ddff4b029a6a1ea09f690052" candidate_uuid: "7e8da74e740d4510a457fb5e6dd1d819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a300244bb80847f68991901ff6ed4f00" is_pre_election: true
I20260812 01:01:59.595614 16249 raft_consensus.cc:2781] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:01:59.595821 15799 leader_election.cc:302] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e8da74e740d4510a457fb5e6dd1d819; no voters: 430435c9b2314d60b15673a369a0c5a3, a300244bb80847f68991901ff6ed4f00
I20260812 01:01:59.596311 15685 raft_consensus.cc:2454] T c0da7326ddff4b029a6a1ea09f690052 P a300244bb80847f68991901ff6ed4f00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 430435c9b2314d60b15673a369a0c5a3 in term 1.
I20260812 01:01:59.596304 16249 raft_consensus.cc:686] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3 [term 1 LEADER]: Becoming Leader. State: Replica: 430435c9b2314d60b15673a369a0c5a3, State: Running, Role: LEADER
I20260812 01:01:59.596571 16244 raft_consensus.cc:2726] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 01:01:59.596824 16241 leader_election.cc:288] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3 [CANDIDATE]: Term 1 election: Requested vote from peers a300244bb80847f68991901ff6ed4f00 (127.21.201.65:35855), 7e8da74e740d4510a457fb5e6dd1d819 (127.21.201.66:34007)
I20260812 01:01:59.597219 16249 consensus_queue.cc:237] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a300244bb80847f68991901ff6ed4f00" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35855 } } peers { permanent_uuid: "430435c9b2314d60b15673a369a0c5a3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34275 } } peers { permanent_uuid: "7e8da74e740d4510a457fb5e6dd1d819" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34007 } }
I20260812 01:01:59.602435 15432 catalog_manager.cc:5495] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 430435c9b2314d60b15673a369a0c5a3 (127.21.201.67). New cstate: current_term: 1 leader_uuid: "430435c9b2314d60b15673a369a0c5a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a300244bb80847f68991901ff6ed4f00" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "430435c9b2314d60b15673a369a0c5a3" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e8da74e740d4510a457fb5e6dd1d819" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34007 } health_report { overall_health: UNKNOWN } } }
I20260812 01:01:59.646592 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:59.640363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38450 (local address 127.21.201.67:34275)
0812 01:01:59.640571 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:59.640577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:59.641175 (+ 598us) server_negotiation.cc:407] Connection header received
0812 01:01:59.641346 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:59.641350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:59.641472 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:59.641674 (+ 202us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:01:59.642042 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.642836 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.644387 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.644778 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.644884 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:59.645421 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:01:59.645975 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:01:59.646095 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:59.646247 (+ 152us) server_negotiation.cc:299] Negotiation successful
0812 01:01:59.646317 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 01:01:59.654853 15685 raft_consensus.cc:1261] T c0da7326ddff4b029a6a1ea09f690052 P a300244bb80847f68991901ff6ed4f00 [term 1 FOLLOWER]: Refusing update from remote peer 430435c9b2314d60b15673a369a0c5a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:01:59.654910 15912 raft_consensus.cc:1261] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819 [term 1 FOLLOWER]: Refusing update from remote peer 430435c9b2314d60b15673a369a0c5a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:01:59.655946 16249 consensus_queue.cc:1035] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e8da74e740d4510a457fb5e6dd1d819" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:01:59.656314 16241 consensus_queue.cc:1035] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a300244bb80847f68991901ff6ed4f00" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:01:59.670593 16265 mvcc.cc:205] Tried to move back new op lower bound from 7317489744495337472 to 7317489744274202624. Current Snapshot: MvccSnapshot[applied={T|T < 7317489744495337472}]
I20260812 01:01:59.680814 15432 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:01:59.681092 15432 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:01:59.684439 15432 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790: Sending DeleteTablet for 3 replicas of tablet c0da7326ddff4b029a6a1ea09f690052
I20260812 01:01:59.685587 15651 tablet_service.cc:1514] Processing DeleteTablet for tablet c0da7326ddff4b029a6a1ea09f690052 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:01:59 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:47396
I20260812 01:01:59.685681 16104 tablet_service.cc:1514] Processing DeleteTablet for tablet c0da7326ddff4b029a6a1ea09f690052 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:01:59 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:38448
I20260812 01:01:59.685853 15873 tablet_service.cc:1514] Processing DeleteTablet for tablet c0da7326ddff4b029a6a1ea09f690052 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:01:59 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:49754
I20260812 01:01:59.688249 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 15497
I20260812 01:01:59.698205 16279 tablet_replica.cc:330] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819: stopping tablet replica
I20260812 01:01:59.698784 16279 raft_consensus.cc:2229] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:01:59.699515 16279 raft_consensus.cc:2258] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:01:59.700608 16277 tablet_replica.cc:330] T c0da7326ddff4b029a6a1ea09f690052 P a300244bb80847f68991901ff6ed4f00: stopping tablet replica
I20260812 01:01:59.700997 16277 raft_consensus.cc:2229] T c0da7326ddff4b029a6a1ea09f690052 P a300244bb80847f68991901ff6ed4f00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:01:59.701452 16277 raft_consensus.cc:2258] T c0da7326ddff4b029a6a1ea09f690052 P a300244bb80847f68991901ff6ed4f00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:01:59.702019 16279 ts_tablet_manager.cc:1905] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:01:59.706264 16277 ts_tablet_manager.cc:1905] T c0da7326ddff4b029a6a1ea09f690052 P a300244bb80847f68991901ff6ed4f00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:01:59.709712 16278 tablet_replica.cc:330] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3: stopping tablet replica
I20260812 01:01:59.710124 16279 ts_tablet_manager.cc:1918] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:01:59.710121 16278 raft_consensus.cc:2229] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3 [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:01:59.710305 16279 log.cc:1196] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/c0da7326ddff4b029a6a1ea09f690052
I20260812 01:01:59.710803 16278 raft_consensus.cc:2258] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:01:59.710901 16279 ts_tablet_manager.cc:1939] T c0da7326ddff4b029a6a1ea09f690052 P 7e8da74e740d4510a457fb5e6dd1d819: Deleting consensus metadata
I20260812 01:01:59.712977 15411 catalog_manager.cc:4841] TS 7e8da74e740d4510a457fb5e6dd1d819 (127.21.201.66:34007): tablet c0da7326ddff4b029a6a1ea09f690052 (table test-table [id=7218cc6a461c47ddb3e4212fd2c7780c]) successfully deleted
I20260812 01:01:59.714814 16278 ts_tablet_manager.cc:1905] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:01:59.718945 16277 ts_tablet_manager.cc:1918] T c0da7326ddff4b029a6a1ea09f690052 P a300244bb80847f68991901ff6ed4f00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:01:59.719235 16277 log.cc:1196] T c0da7326ddff4b029a6a1ea09f690052 P a300244bb80847f68991901ff6ed4f00: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/wal/wals/c0da7326ddff4b029a6a1ea09f690052
I20260812 01:01:59.719861 16277 ts_tablet_manager.cc:1939] T c0da7326ddff4b029a6a1ea09f690052 P a300244bb80847f68991901ff6ed4f00: Deleting consensus metadata
I20260812 01:01:59.722039 15414 catalog_manager.cc:4841] TS a300244bb80847f68991901ff6ed4f00 (127.21.201.65:35855): tablet c0da7326ddff4b029a6a1ea09f690052 (table test-table [id=7218cc6a461c47ddb3e4212fd2c7780c]) successfully deleted
I20260812 01:01:59.722740 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:59.689606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47398 (local address 127.21.201.65:35855)
0812 01:01:59.689805 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:59.689811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:59.695718 (+ 5907us) server_negotiation.cc:407] Connection header received
0812 01:01:59.696953 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:59.696962 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:59.697077 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:59.697171 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:59.701862 (+ 4691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.702730 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.706644 (+ 3914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.707587 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.707719 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:59.714198 (+ 6479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:59.714213 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:59.714230 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:59.714293 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:59.718754 (+ 4461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:59.719862 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:59.719871 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:59.719875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:59.720057 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:59.720566 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:59.720571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:59.720575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:59.721089 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:59.721456 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:59.722101 (+ 645us) server_negotiation.cc:299] Negotiation successful
0812 01:01:59.722365 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 01:01:59.723249 16278 ts_tablet_manager.cc:1918] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:01:59.723487 16278 log.cc:1196] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/c0da7326ddff4b029a6a1ea09f690052
I20260812 01:01:59.723855 15753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:01:59.724248 16278 ts_tablet_manager.cc:1939] T c0da7326ddff4b029a6a1ea09f690052 P 430435c9b2314d60b15673a369a0c5a3: Deleting consensus metadata
I20260812 01:01:59.727682 15411 catalog_manager.cc:4841] TS 430435c9b2314d60b15673a369a0c5a3 (127.21.201.67:34275): tablet c0da7326ddff4b029a6a1ea09f690052 (table test-table [id=7218cc6a461c47ddb3e4212fd2c7780c]) successfully deleted
I20260812 01:01:59.832835 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 15497
I20260812 01:01:59.859853 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 15766
I20260812 01:01:59.877625 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:01:59.860720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49762 (local address 127.21.201.66:34007)
0812 01:01:59.860906 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 01:01:59.860911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:01:59.860994 (+ 83us) server_negotiation.cc:407] Connection header received
0812 01:01:59.861177 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:01:59.861181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:01:59.861248 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:01:59.861337 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:01:59.863136 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.863983 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.865940 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:01:59.866366 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:01:59.866476 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:01:59.870146 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:01:59.870159 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:01:59.870175 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:01:59.870213 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:01:59.873947 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:59.874712 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:59.874723 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:59.874726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:59.874903 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:01:59.875319 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:01:59.875325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:01:59.875328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:01:59.875865 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:01:59.876189 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0812 01:01:59.876661 (+ 472us) server_negotiation.cc:299] Negotiation successful
0812 01:01:59.877042 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 01:01:59.878437 15982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:01:59.980625 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 15766
I20260812 01:02:00.009977 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 15995
I20260812 01:02:00.030012 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:00.011264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38462 (local address 127.21.201.67:34275)
0812 01:02:00.011509 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:00.011517 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:00.011547 (+ 30us) server_negotiation.cc:407] Connection header received
0812 01:02:00.011629 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:00.011636 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:00.011727 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:00.011890 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:02:00.013716 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:00.014927 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:00.017065 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:00.017824 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:00.017989 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:00.021517 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:02:00.021532 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:02:00.021554 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:02:00.021599 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:02:00.026155 (+ 4556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:00.027203 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:00.027215 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:00.027219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:00.027379 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:00.027900 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:00.027909 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:00.027913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:00.028487 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:02:00.028833 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:00.029194 (+ 361us) server_negotiation.cc:299] Negotiation successful
0812 01:02:00.029466 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 01:02:00.030906 16197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:02:00.163462 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 15995
I20260812 01:02:00.190933 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 15388
I20260812 01:02:00.192173 15486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:02:00.310096 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 15388
I20260812 01:02:00.345619 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:46059
--webserver_interface=127.21.201.126
--webserver_port=41783
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:46059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:02:00.714251 16292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:02:00.714684 16292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:02:00.714864 16292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:02:00.725469 16292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:02:00.725589 16292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:02:00.725637 16292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:02:00.725700 16292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:02:00.725745 16292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:02:00.735996 16292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:46059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:46059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=41783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:02:00.737357 16292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:02:00.738658 16292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:02:00.750917 16299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:02:00.750833 16300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:02:00.750864 16303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:02:00.751626 16292 server_base.cc:1034] running on GCE node
I20260812 01:02:00.752497 16292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:02:00.753988 16292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:02:00.755236 16292 hybrid_clock.cc:648] HybridClock initialized: now 1786496520755186 us; error 38 us; skew 500 ppm
I20260812 01:02:00.755631 16292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:02:00 dist-test-slave-7f01 krb5kdc[14409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496520, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:02:00.759071 16292 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 01:02:00.760711 16292 webserver.cc:460] Webserver started at http://127.21.201.126:41783/ using document root <none> and password file <none>
I20260812 01:02:00.761456 16292 fs_manager.cc:359] Metadata directory not provided
I20260812 01:02:00.761615 16292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:02:00.767205 16292 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260812 01:02:00.770285 16318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:00.771597 16292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 01:02:00.771854 16292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "86b526280972413ebeff882ad481b790"
format_stamp: "Formatted at 2026-08-12 01:01:53 on dist-test-slave-7f01"
server_key: "af406f1ffed7fbf9f7baf5a90da6c526"
server_key_iv: "65375d5c4608cf8ef067171a16a0f6aa"
server_key_version: "encryptionkey@0"
I20260812 01:02:00.772886 16292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:02:00.788568 16292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:02:00.792656 16292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:02:00.792948 16292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:02:00.812721 16292 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:46059
I20260812 01:02:00.812741 16406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:46059 every 8 connection(s)
I20260812 01:02:00.814368 16292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:02:00.821396 16407 sys_catalog.cc:263] Verifying existing consensus state
I20260812 01:02:00.821739 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 16292
I20260812 01:02:00.822604 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:35855
--local_ip_for_outbound_sockets=127.21.201.65
--tserver_master_addrs=127.21.201.126:46059
--webserver_port=34645
--webserver_interface=127.21.201.65
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:02:00.824999 16407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790: Bootstrap starting.
I20260812 01:02:00.848924 16407 log.cc:824] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790: Log is configured to *not* fsync() on all Append() calls
I20260812 01:02:00.865863 16407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 01:02:00.866463 16407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790: Bootstrap complete.
I20260812 01:02:00.876317 16407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b526280972413ebeff882ad481b790" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46059 } }
I20260812 01:02:00.877453 16407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b526280972413ebeff882ad481b790, State: Initialized, Role: FOLLOWER
I20260812 01:02:00.878129 16407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b526280972413ebeff882ad481b790" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46059 } }
I20260812 01:02:00.878376 16407 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:02:00.878505 16407 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:02:00.878716 16407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [term 2 FOLLOWER]: Advancing to term 3
I20260812 01:02:00.882745 16407 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b526280972413ebeff882ad481b790" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46059 } }
I20260812 01:02:00.883265 16407 leader_election.cc:302] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86b526280972413ebeff882ad481b790; no voters:
I20260812 01:02:00.884032 16407 leader_election.cc:288] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 01:02:00.884183 16413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 01:02:00.885555 16413 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [term 3 LEADER]: Becoming Leader. State: Replica: 86b526280972413ebeff882ad481b790, State: Running, Role: LEADER
I20260812 01:02:00.886241 16413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b526280972413ebeff882ad481b790" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46059 } }
I20260812 01:02:00.886463 16407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:02:00.889555 16415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86b526280972413ebeff882ad481b790. Latest consensus state: current_term: 3 leader_uuid: "86b526280972413ebeff882ad481b790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b526280972413ebeff882ad481b790" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46059 } } }
I20260812 01:02:00.890008 16415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [sys.catalog]: This master's current role is: LEADER
I20260812 01:02:00.890632 16414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "86b526280972413ebeff882ad481b790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b526280972413ebeff882ad481b790" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46059 } } }
I20260812 01:02:00.890946 16414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86b526280972413ebeff882ad481b790 [sys.catalog]: This master's current role is: LEADER
I20260812 01:02:00.893260 16423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:02:00.905339 16423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:02:00.907186 16423 catalog_manager.cc:1259] Loaded cluster ID: 9f307e7a184d412fa2314b73e672dead
I20260812 01:02:00.907824 16423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260812 01:02:00.910524 16423 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f763c09b9ad google::LogMessage::Fail() at ??:0
@ 0x7f763c09e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f763c09b4de google::LogMessage::Flush() at ??:0
@ 0x7f763c09ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7651ace512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f7651acb976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7651b53610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f7651b53439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f763ece7345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f763ecea91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f763ecea799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f763ecc5eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7641f9f6db start_thread at ??:0
@ 0x7f763af3561f clone at ??:0
W20260812 01:02:01.271859 16411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:02:01.272174 16411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:02:01.272390 16411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:02:01.279979 16411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:02:01.280112 16411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:02:01.280308 16411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 01:02:01.290402 16411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:35855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=34645
--tserver_master_addrs=127.21.201.126:46059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:02:01.291752 16411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:02:01.293136 16411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:02:01.306999 16479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:02:01.306948 16482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:02:01.307070 16484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:02:01.308035 16411 server_base.cc:1034] running on GCE node
I20260812 01:02:01.308746 16411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:02:01.309725 16411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:02:01.310946 16411 hybrid_clock.cc:648] HybridClock initialized: now 1786496521310895 us; error 29 us; skew 500 ppm
I20260812 01:02:01.311270 16411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:02:01 dist-test-slave-7f01 krb5kdc[14409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496521, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:02:01.314697 16411 init.cc:377] Logged in from keytab as kudu/127.21.201.65@KRBTEST.COM (short username kudu)
I20260812 01:02:01.316308 16411 webserver.cc:460] Webserver started at http://127.21.201.65:34645/ using document root <none> and password file <none>
I20260812 01:02:01.317123 16411 fs_manager.cc:359] Metadata directory not provided
I20260812 01:02:01.317265 16411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:02:01.322561 16411 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.004s
I20260812 01:02:01.325235 16497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:01.326537 16411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260812 01:02:01.326709 16411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "a300244bb80847f68991901ff6ed4f00"
format_stamp: "Formatted at 2026-08-12 01:01:54 on dist-test-slave-7f01"
server_key: "dd08d3277f3e90779b064b315e917363"
server_key_iv: "1d4825eee7ddf41a0f3d17287fdd13b4"
server_key_version: "encryptionkey@0"
I20260812 01:02:01.327826 16411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:02:01.352149 16411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:02:01.356130 16411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:02:01.356427 16411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:02:01.357511 16411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:02:01.359035 16411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:02:01.359171 16411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:01.359308 16411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:02:01.359413 16411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:01.395982 16411 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:35855
I20260812 01:02:01.396037 16698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:35855 every 8 connection(s)
I20260812 01:02:01.397490 16411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 01:02:01.398685 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 16411
I20260812 01:02:01.399490 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:34007
--local_ip_for_outbound_sockets=127.21.201.66
--tserver_master_addrs=127.21.201.126:46059
--webserver_port=36363
--webserver_interface=127.21.201.66
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:02:01.402923 16704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:01.400825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.65:59613)
0812 01:02:01.401709 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:01.402031 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":374,"threads_started":1}
W20260812 01:02:01.403700 16699 heartbeater.cc:644] Failed to heartbeat to 127.21.201.126:46059 (0 consecutive failures): Network error: Failed to ping master at 127.21.201.126:46059: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
I20260812 01:02:01.404647 16704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:01.404281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.65:54137)
0812 01:02:01.404439 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:01.404533 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260812 01:02:01.405649 16704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:01.405310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.65:60207)
0812 01:02:01.405454 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:01.405535 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 01:02:01.405963 16699 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 01:02:01.753046 16708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:02:01.753515 16708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:02:01.753629 16708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:02:01.761969 16708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:02:01.762125 16708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:02:01.762253 16708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 01:02:01.773877 16708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:34007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=36363
--tserver_master_addrs=127.21.201.126:46059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:02:01.775243 16708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:02:01.776747 16708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:02:01.790606 16714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:02:01.790701 16716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:02:01.790735 16713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:02:01.791360 16708 server_base.cc:1034] running on GCE node
I20260812 01:02:01.792084 16708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:02:01.793033 16708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:02:01.794271 16708 hybrid_clock.cc:648] HybridClock initialized: now 1786496521794233 us; error 51 us; skew 500 ppm
I20260812 01:02:01.794598 16708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:02:01 dist-test-slave-7f01 krb5kdc[14409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496521, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:02:01.797852 16708 init.cc:377] Logged in from keytab as kudu/127.21.201.66@KRBTEST.COM (short username kudu)
I20260812 01:02:01.799309 16708 webserver.cc:460] Webserver started at http://127.21.201.66:36363/ using document root <none> and password file <none>
I20260812 01:02:01.800102 16708 fs_manager.cc:359] Metadata directory not provided
I20260812 01:02:01.800261 16708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:02:01.805787 16708 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 01:02:01.808658 16732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:01.810012 16708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:02:01.810205 16708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "7e8da74e740d4510a457fb5e6dd1d819"
format_stamp: "Formatted at 2026-08-12 01:01:55 on dist-test-slave-7f01"
server_key: "1f2696dfc83d5691f9e1af5b3585ba38"
server_key_iv: "f8e645c442be961a09a5b5c1b6bf1b32"
server_key_version: "encryptionkey@0"
I20260812 01:02:01.811285 16708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:02:01.831300 16708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:02:01.836359 16708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:02:01.836696 16708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:02:01.838189 16708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:02:01.840440 16708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:02:01.840756 16708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:01.841023 16708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:02:01.841182 16708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:01.899818 16708 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:34007
I20260812 01:02:01.899885 16910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:34007 every 8 connection(s)
I20260812 01:02:01.901536 16708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 01:02:01.907298 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:01.905262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.66:52531)
0812 01:02:01.906128 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:01.906362 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":357,"threads_started":1}
W20260812 01:02:01.908257 16911 heartbeater.cc:644] Failed to heartbeat to 127.21.201.126:46059 (0 consecutive failures): Network error: Failed to ping master at 127.21.201.126:46059: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
I20260812 01:02:01.909303 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:01.908901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.66:37651)
0812 01:02:01.909109 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:01.909186 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260812 01:02:01.910830 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:01.910102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.66:54655)
0812 01:02:01.910296 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:01.910607 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260812 01:02:01.911489 16911 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 01:02:01.911948 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 16708
I20260812 01:02:01.912660 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:34275
--local_ip_for_outbound_sockets=127.21.201.67
--tserver_master_addrs=127.21.201.126:46059
--webserver_port=38743
--webserver_interface=127.21.201.67
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:02:02.314661 16915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:02:02.315158 16915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:02:02.315277 16915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:02:02.324884 16915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:02:02.325076 16915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:02:02.325218 16915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 01:02:02.338640 16915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:34275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=38743
--tserver_master_addrs=127.21.201.126:46059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:02:02.340780 16915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:02:02.342464 16915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:02:02.359591 16923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:02:02.359591 16925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:02:02.359895 16930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:02:02.360560 16915 server_base.cc:1034] running on GCE node
I20260812 01:02:02.361315 16915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:02:02.362464 16915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:02:02.363998 16915 hybrid_clock.cc:648] HybridClock initialized: now 1786496522363926 us; error 56 us; skew 500 ppm
I20260812 01:02:02.364347 16915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:02:02 dist-test-slave-7f01 krb5kdc[14409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496522, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:02:02.374009 16915 init.cc:377] Logged in from keytab as kudu/127.21.201.67@KRBTEST.COM (short username kudu)
I20260812 01:02:02.376485 16915 webserver.cc:460] Webserver started at http://127.21.201.67:38743/ using document root <none> and password file <none>
I20260812 01:02:02.377702 16915 fs_manager.cc:359] Metadata directory not provided
I20260812 01:02:02.377910 16915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:02:02.386231 16915 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 01:02:02.391414 16940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:02.393446 16915 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260812 01:02:02.393790 16915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "430435c9b2314d60b15673a369a0c5a3"
format_stamp: "Formatted at 2026-08-12 01:01:55 on dist-test-slave-7f01"
server_key: "a328c36199471f699ec4562866f8f5aa"
server_key_iv: "16e8525042db1c138c59b8a98926fc4e"
server_key_version: "encryptionkey@0"
I20260812 01:02:02.394904 16915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:02:02.408165 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:02.406913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.65:36789)
0812 01:02:02.407596 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:02.407863 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":353,"threads_started":1}
I20260812 01:02:02.422632 16915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:02:02.428179 16915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:02:02.428511 16915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:02:02.429730 16915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:02:02.431550 16915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:02:02.431700 16915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:02.431871 16915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:02:02.431998 16915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:02.475347 16915 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:34275
I20260812 01:02:02.475407 17122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:34275 every 8 connection(s)
I20260812 01:02:02.476943 16915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 01:02:02.481734 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 16915
I20260812 01:02:02.482605 17125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:02.480205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.67:51189)
0812 01:02:02.481022 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:02.481377 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":331,"threads_started":1}
W20260812 01:02:02.483826 17123 heartbeater.cc:644] Failed to heartbeat to 127.21.201.126:46059 (0 consecutive failures): Network error: Failed to ping master at 127.21.201.126:46059: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
I20260812 01:02:02.485103 17125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:02.484667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.67:58015)
0812 01:02:02.484859 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:02.484943 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260812 01:02:02.486251 17125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:02.485858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.67:46211)
0812 01:02:02.486035 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:02.486126 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260812 01:02:02.486676 17123 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 01:02:02.912936 17128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:02.912090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.66:57399)
0812 01:02:02.912560 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:02.912700 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":225,"threads_started":1}
I20260812 01:02:03.409927 17131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:03.409176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.65:43747)
0812 01:02:03.409657 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:03.409746 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":240,"threads_started":1}
I20260812 01:02:03.488081 17132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:03.487200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.67:36513)
0812 01:02:03.487661 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:03.487840 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":232,"threads_started":1}
I20260812 01:02:03.914656 17133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:03.913905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.66:54677)
0812 01:02:03.914405 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:03.914489 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":277,"threads_started":1}
I20260812 01:02:04.411598 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:04.410855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.65:45691)
0812 01:02:04.411320 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:04.411404 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":242,"threads_started":1}
I20260812 01:02:04.489764 17135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:04.488985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.67:59631)
0812 01:02:04.489512 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:04.489593 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":300,"threads_started":1}
I20260812 01:02:04.916435 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:04.915587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.66:41231)
0812 01:02:04.916175 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:04.916258 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":401,"threads_started":1}
I20260812 01:02:05.413471 17139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:05.412584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.65:60671)
0812 01:02:05.413093 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:05.413230 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":284,"threads_started":1}
I20260812 01:02:05.491662 17140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:05.490790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.67:53335)
0812 01:02:05.491364 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:05.491470 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":334,"threads_started":1}
I20260812 01:02:05.918164 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:05.917407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.66:52955)
0812 01:02:05.917897 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:05.917979 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":222,"threads_started":1}
I20260812 01:02:06.415623 17142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:06.414679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.65:56067)
0812 01:02:06.415341 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:06.415438 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":394,"threads_started":1}
I20260812 01:02:06.493359 17143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:06.492595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.67:40729)
0812 01:02:06.493084 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:06.493169 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":265,"threads_started":1}
I20260812 01:02:06.919868 17145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:06.919044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.66:38511)
0812 01:02:06.919527 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:06.919610 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":254,"threads_started":1}
I20260812 01:02:07.417701 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:07.416650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.65:42567)
0812 01:02:07.417186 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:07.417333 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":263,"threads_started":1}
I20260812 01:02:07.495280 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:07.494319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.67:36193)
0812 01:02:07.494901 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:07.495032 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":304,"threads_started":1}
I20260812 01:02:07.921396 17149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:07.920696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.66:38489)
0812 01:02:07.921135 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:07.921225 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":231,"threads_started":1}
I20260812 01:02:08.419497 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:08.418652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.65:40137)
0812 01:02:08.419181 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:08.419268 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":239,"threads_started":1}
I20260812 01:02:08.497040 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:08.496306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.67:50777)
0812 01:02:08.496746 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:08.496838 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":230,"threads_started":1}
I20260812 01:02:08.923116 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:08.922280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.66:46495)
0812 01:02:08.922797 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:08.922888 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":218,"threads_started":1}
I20260812 01:02:09.421346 17161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:09.420579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.65:40041)
0812 01:02:09.421076 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:09.421175 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":233,"threads_started":1}
I20260812 01:02:09.499027 17162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:09.497993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.67:47533)
0812 01:02:09.498679 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:09.498761 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":425,"threads_started":1}
I20260812 01:02:09.924726 17164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:09.924021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.66:60011)
0812 01:02:09.924457 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:09.924552 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":223,"threads_started":1}
I20260812 01:02:10.422981 17165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:10.422197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.65:50121)
0812 01:02:10.422693 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:10.422787 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":243,"threads_started":1}
I20260812 01:02:10.500768 17166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:10.500019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.67:49115)
0812 01:02:10.500508 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:10.500591 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":272,"threads_started":1}
I20260812 01:02:10.926746 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:10.925726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.66:47731)
0812 01:02:10.926471 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:10.926553 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":385,"threads_started":1}
I20260812 01:02:11.424984 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:11.424102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.65:60953)
0812 01:02:11.424610 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:11.424727 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":265,"threads_started":1}
I20260812 01:02:11.502673 17172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:11.501691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.67:52725)
0812 01:02:11.502266 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:11.502420 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":311,"threads_started":1}
I20260812 01:02:11.928783 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:11.928068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.66:36219)
0812 01:02:11.928516 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:11.928607 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":239,"threads_started":1}
I20260812 01:02:12.426788 17174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:12.425929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.65:47863)
0812 01:02:12.426481 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:12.426584 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":303,"threads_started":1}
I20260812 01:02:12.504423 17175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:12.503598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.67:51931)
0812 01:02:12.504168 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:12.504250 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":360,"threads_started":1}
I20260812 01:02:12.930454 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:12.929660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.66:50445)
0812 01:02:12.930156 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:12.930255 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":259,"threads_started":1}
I20260812 01:02:13.428562 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:13.427810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.65:36779)
0812 01:02:13.428284 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:13.428372 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":248,"threads_started":1}
I20260812 01:02:13.506162 17178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:13.505261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.67:45753)
0812 01:02:13.505715 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:13.505846 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":212,"threads_started":1}
I20260812 01:02:13.932179 17180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:13.931379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.66:55287)
0812 01:02:13.931931 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:13.932016 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":280,"threads_started":1}
I20260812 01:02:14.430411 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:14.429524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.65:39981)
0812 01:02:14.430070 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:14.430200 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":278,"threads_started":1}
I20260812 01:02:14.508069 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:14.507253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.67:58191)
0812 01:02:14.507810 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:14.507891 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":285,"threads_started":1}
I20260812 01:02:14.933854 17185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:14.933064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.66:45595)
0812 01:02:14.933530 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:14.933642 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":235,"threads_started":1}
I20260812 01:02:15.432392 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:15.431407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.65:55667)
0812 01:02:15.432055 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:15.432139 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":351,"threads_started":1}
I20260812 01:02:15.509758 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:15.508960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.67:43761)
0812 01:02:15.509444 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:15.509557 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":237,"threads_started":1}
I20260812 01:02:15.935428 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:15.934734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.66:39597)
0812 01:02:15.935155 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:15.935237 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":211,"threads_started":1}
I20260812 01:02:16.434450 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:16.433569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.65:42965)
0812 01:02:16.434128 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:16.434257 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":299,"threads_started":1}
I20260812 01:02:16.511660 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:16.510777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.67:57009)
0812 01:02:16.511331 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:16.511427 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":315,"threads_started":1}
I20260812 01:02:16.937408 17193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:16.936571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.66:37375)
0812 01:02:16.937097 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:16.937211 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":266,"threads_started":1}
I20260812 01:02:17.436512 17194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:17.435542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.65:41663)
0812 01:02:17.436243 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:17.436341 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":435,"threads_started":1}
I20260812 01:02:17.494015 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 16411
I20260812 01:02:17.513763 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:17.512975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.67:47485)
0812 01:02:17.513448 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:17.513570 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":226,"threads_started":1}
I20260812 01:02:17.939200 17198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:17.938422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46059 (local address 127.21.201.66:39147)
0812 01:02:17.938913 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:17.939031 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:46059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":248,"threads_started":1}
I20260812 01:02:18.121680 17195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:17.495344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42676 (local address 127.21.201.65:35855)
0812 01:02:17.496379 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:17.496403 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:18.104515 (+608112us) server_negotiation.cc:407] Connection header received
0812 01:02:18.106269 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:18.106280 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:18.106776 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:18.107149 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:02:18.108778 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:18.110172 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:18.111582 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:18.112387 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:18.112528 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:18.115380 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:02:18.115387 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:02:18.115402 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:02:18.115446 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:02:18.118627 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:18.119298 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:18.119304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:18.119311 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:18.119460 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:18.119819 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:18.119827 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:18.119829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:18.120354 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:02:18.120595 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:18.120927 (+ 332us) server_negotiation.cc:299] Negotiation successful
0812 01:02:18.121366 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":768,"thread_start_us":563,"threads_started":1}
I20260812 01:02:18.123076 16691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:02:18.210251 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 16411
I20260812 01:02:18.239166 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 16708
I20260812 01:02:18.257174 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:18.240364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43660 (local address 127.21.201.66:34007)
0812 01:02:18.241012 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:18.241026 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:18.241070 (+ 44us) server_negotiation.cc:407] Connection header received
0812 01:02:18.242836 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:18.242846 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:18.243266 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:18.243560 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:02:18.245054 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:18.246298 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:18.247634 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:18.248381 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:18.248531 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:18.251171 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:02:18.251178 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:02:18.251192 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:02:18.251226 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:02:18.254033 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:18.254638 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:18.254644 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:18.254649 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:18.254800 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:18.255112 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:18.255115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:18.255117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:18.255629 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:02:18.255943 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:18.256429 (+ 486us) server_negotiation.cc:299] Negotiation successful
0812 01:02:18.256863 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":283,"threads_started":1}
I20260812 01:02:18.258563 16905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:02:18.345104 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 16708
I20260812 01:02:18.372076 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 16915
I20260812 01:02:18.390201 17205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:18.373098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55146 (local address 127.21.201.67:34275)
0812 01:02:18.373617 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:18.373631 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:18.373669 (+ 38us) server_negotiation.cc:407] Connection header received
0812 01:02:18.375223 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:18.375232 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:18.375642 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:18.376045 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:02:18.377623 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:18.378891 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:18.380253 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:18.380931 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:18.381147 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:18.383895 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:02:18.383904 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:02:18.383917 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:02:18.383962 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:02:18.386842 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:18.387585 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:18.387592 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:18.387597 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:18.387819 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:18.388204 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:18.388209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:18.388213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:18.388727 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:02:18.389055 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:18.389477 (+ 422us) server_negotiation.cc:299] Negotiation successful
0812 01:02:18.389897 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":246,"threads_started":1}
I20260812 01:02:18.391568 17115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:02:18.479108 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 16915
2026-08-12T01:02:18Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (26658 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:02:18 dist-test-slave-7f01 krb5kdc[17209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:02:18 dist-test-slave-7f01 krb5kdc[17209](info): set up 2 sockets
Aug 12 01:02:18 dist-test-slave-7f01 krb5kdc[17209](info): commencing operation
krb5kdc: starting...
W20260812 01:02:19.876561 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.348s user 0.005s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:02:20 dist-test-slave-7f01 krb5kdc[17209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:02:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:02:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 01:02:20.204437 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:32907
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:39133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:32907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:02:20.567283 17232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:02:20.567592 17232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:02:20.567691 17232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:02:20.575034 17232 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 01:02:20.575124 17232 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 01:02:20.575165 17232 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 01:02:20.575220 17232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:02:20.575282 17232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:02:20.575348 17232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:02:20.575389 17232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:02:20.575440 17232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:02:20.585485 17232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:39133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:32907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:32907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:02:20.586814 17232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:02:20.588230 17232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:02:20.601667 17242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:02:20.601869 17247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:02:20.601716 17240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:02:20.602679 17232 server_base.cc:1034] running on GCE node
I20260812 01:02:20.603713 17232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:02:20.605505 17232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:02:20.606802 17232 hybrid_clock.cc:648] HybridClock initialized: now 1786496540606706 us; error 66 us; skew 500 ppm
I20260812 01:02:20.607194 17232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:02:20 dist-test-slave-7f01 krb5kdc[17209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496540, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:02:20.611181 17232 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 01:02:20.613013 17232 webserver.cc:460] Webserver started at http://127.21.201.126:38777/ using document root <none> and password file <none>
I20260812 01:02:20.613788 17232 fs_manager.cc:359] Metadata directory not provided
I20260812 01:02:20.613938 17232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:02:20.614279 17232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:02:20.617579 17232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "1409835d19b84f5c84fef34e4cc8e8d3"
format_stamp: "Formatted at 2026-08-12 01:02:20 on dist-test-slave-7f01"
server_key: "e2b99432c1792435d918d9268787e6de"
server_key_iv: "25f718051a7f4176395a81a6be28962f"
server_key_version: "encryptionkey@0"
I20260812 01:02:20.618464 17232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "1409835d19b84f5c84fef34e4cc8e8d3"
format_stamp: "Formatted at 2026-08-12 01:02:20 on dist-test-slave-7f01"
server_key: "e2b99432c1792435d918d9268787e6de"
server_key_iv: "25f718051a7f4176395a81a6be28962f"
server_key_version: "encryptionkey@0"
I20260812 01:02:20.625224 17232 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 01:02:20.629778 17259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:20.631378 17232 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 01:02:20.631595 17232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "1409835d19b84f5c84fef34e4cc8e8d3"
format_stamp: "Formatted at 2026-08-12 01:02:20 on dist-test-slave-7f01"
server_key: "e2b99432c1792435d918d9268787e6de"
server_key_iv: "25f718051a7f4176395a81a6be28962f"
server_key_version: "encryptionkey@0"
I20260812 01:02:20.631893 17232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:02:20.653286 17232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:02:20.657495 17232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:02:20.657729 17232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:02:20.677397 17232 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:32907
I20260812 01:02:20.677399 17351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:32907 every 8 connection(s)
I20260812 01:02:20.678828 17232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:02:20.682827 17352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:02:20.684055 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 17232
I20260812 01:02:20.684312 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:02:20.685256 22309 external_mini_cluster.cc:1428] Setting key c893be18eb530e1ff332f30cadadccf4
I20260812 01:02:20.698426 17352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3: Bootstrap starting.
Aug 12 01:02:20 dist-test-slave-7f01 krb5kdc[17209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:02:20.702241 17352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3: Neither blocks nor log segments found. Creating new log.
I20260812 01:02:20.703606 17352 log.cc:824] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3: Log is configured to *not* fsync() on all Append() calls
I20260812 01:02:20.707046 17352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3: No bootstrap required, opened a new log
I20260812 01:02:20.710661 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:20.687585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51420 (local address 127.21.201.126:32907)
0812 01:02:20.688521 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:20.688537 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:20.688581 (+ 44us) server_negotiation.cc:407] Connection header received
0812 01:02:20.690244 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:20.690267 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:20.690654 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:20.691004 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:02:20.693020 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:20.694314 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:20.696257 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:20.696894 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:20.697052 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:20.702092 (+ 5040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:02:20.702100 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:02:20.702115 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:02:20.702166 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:02:20.705811 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:20.706808 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:20.706815 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:20.706822 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:20.707039 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:20.707513 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:20.707518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:20.707521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:20.708174 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:02:20.708504 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:20.709093 (+ 589us) server_negotiation.cc:299] Negotiation successful
0812 01:02:20.709585 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":485,"threads_started":1}
I20260812 01:02:20.716852 17352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1409835d19b84f5c84fef34e4cc8e8d3" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 32907 } }
I20260812 01:02:20.717289 17352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:02:20.717412 17352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1409835d19b84f5c84fef34e4cc8e8d3, State: Initialized, Role: FOLLOWER
I20260812 01:02:20.717988 17352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1409835d19b84f5c84fef34e4cc8e8d3" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 32907 } }
I20260812 01:02:20.718261 17352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:02:20.718416 17352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:02:20.718595 17352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:02:20.722138 17352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1409835d19b84f5c84fef34e4cc8e8d3" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 32907 } }
I20260812 01:02:20.722898 17352 leader_election.cc:302] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1409835d19b84f5c84fef34e4cc8e8d3; no voters:
I20260812 01:02:20.724058 17352 leader_election.cc:288] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:02:20.724220 17357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:02:20.725143 17357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [term 1 LEADER]: Becoming Leader. State: Replica: 1409835d19b84f5c84fef34e4cc8e8d3, State: Running, Role: LEADER
I20260812 01:02:20.725848 17357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1409835d19b84f5c84fef34e4cc8e8d3" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 32907 } }
I20260812 01:02:20.727043 17352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:02:20.729409 17364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1409835d19b84f5c84fef34e4cc8e8d3. Latest consensus state: current_term: 1 leader_uuid: "1409835d19b84f5c84fef34e4cc8e8d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1409835d19b84f5c84fef34e4cc8e8d3" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 32907 } } }
I20260812 01:02:20.729883 17364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [sys.catalog]: This master's current role is: LEADER
I20260812 01:02:20.729401 17358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1409835d19b84f5c84fef34e4cc8e8d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1409835d19b84f5c84fef34e4cc8e8d3" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 32907 } } }
I20260812 01:02:20.730437 17358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [sys.catalog]: This master's current role is: LEADER
I20260812 01:02:20.733063 17370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:02:20.739845 17370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:02:20.751083 17370 catalog_manager.cc:1347] Generated new cluster ID: 746586bb7f194fd3ac33abe2533067a3
I20260812 01:02:20.751232 17370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:02:20.765050 17370 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:02:20.767581 17370 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:02:20.781445 17370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3: Generated new TSK 0
I20260812 01:02:20.782653 17370 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.201.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
I20260812 01:02:20.864483 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:0
--local_ip_for_outbound_sockets=127.21.201.65
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:32907
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:39133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:02:21.243816 17394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:02:21.244128 17394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:02:21.244297 17394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:02:21.251636 17394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:02:21.251780 17394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:02:21.251971 17394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 01:02:21.261734 17394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:39133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:32907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:02:21.263057 17394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:02:21.264456 17394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:02:21.278409 17408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:02:21.278437 17411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:02:21.278522 17414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:02:21.279191 17394 server_base.cc:1034] running on GCE node
I20260812 01:02:21.279838 17394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:02:21.281018 17394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:02:21.282284 17394 hybrid_clock.cc:648] HybridClock initialized: now 1786496541282224 us; error 57 us; skew 500 ppm
I20260812 01:02:21.282635 17394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:02:21 dist-test-slave-7f01 krb5kdc[17209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496541, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:02:21.286746 17394 init.cc:377] Logged in from keytab as kudu/127.21.201.65@KRBTEST.COM (short username kudu)
I20260812 01:02:21.288694 17394 webserver.cc:460] Webserver started at http://127.21.201.65:33643/ using document root <none> and password file <none>
I20260812 01:02:21.289482 17394 fs_manager.cc:359] Metadata directory not provided
I20260812 01:02:21.289639 17394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:02:21.290009 17394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:02:21.293313 17394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/data/instance:
uuid: "1c6362753ad54842abda04607ea7cbd5"
format_stamp: "Formatted at 2026-08-12 01:02:21 on dist-test-slave-7f01"
server_key: "8616e700ee2419079e083760906ada76"
server_key_iv: "ed27dcd2203d8f5567c847f828c40a1e"
server_key_version: "encryptionkey@0"
I20260812 01:02:21.294148 17394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance:
uuid: "1c6362753ad54842abda04607ea7cbd5"
format_stamp: "Formatted at 2026-08-12 01:02:21 on dist-test-slave-7f01"
server_key: "8616e700ee2419079e083760906ada76"
server_key_iv: "ed27dcd2203d8f5567c847f828c40a1e"
server_key_version: "encryptionkey@0"
I20260812 01:02:21.299809 17394 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260812 01:02:21.303339 17426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:21.304689 17394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 01:02:21.304888 17394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "1c6362753ad54842abda04607ea7cbd5"
format_stamp: "Formatted at 2026-08-12 01:02:21 on dist-test-slave-7f01"
server_key: "8616e700ee2419079e083760906ada76"
server_key_iv: "ed27dcd2203d8f5567c847f828c40a1e"
server_key_version: "encryptionkey@0"
I20260812 01:02:21.305116 17394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:02:21.320504 17394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:02:21.324390 17394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:02:21.324681 17394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:02:21.325626 17394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:02:21.327097 17394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:02:21.327198 17394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:21.327350 17394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:02:21.327442 17394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:21.366488 17394 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:41747
I20260812 01:02:21.366539 17620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:41747 every 8 connection(s)
I20260812 01:02:21.367949 17394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 01:02:21.369837 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 17394
I20260812 01:02:21.370203 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance
I20260812 01:02:21.371456 22309 external_mini_cluster.cc:1428] Setting key ac3ccd2ac40e332db4221d4aba40f05c
Aug 12 01:02:21 dist-test-slave-7f01 krb5kdc[17209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496541, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:02:21.395197 17623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:21.372144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:35795 (local address 127.21.201.126:32907)
0812 01:02:21.372791 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:21.372798 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:21.373871 (+ 1073us) server_negotiation.cc:407] Connection header received
0812 01:02:21.376784 (+ 2913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:21.376789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:21.376858 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:21.376967 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:02:21.379245 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:21.380104 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:21.382732 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:21.383169 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:21.383278 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:21.388474 (+ 5196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:02:21.388480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:02:21.388484 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:02:21.388520 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:02:21.391361 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:21.392482 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:21.392486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:21.392489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:21.392623 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:21.393167 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:21.393170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:21.393173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:21.393469 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:02:21.393590 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:21.394695 (+ 1105us) server_negotiation.cc:299] Negotiation successful
0812 01:02:21.394902 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":417,"threads_started":1}
I20260812 01:02:21.397177 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:21.372749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.65:35795)
0812 01:02:21.373683 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:21.373736 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 01:02:21.376447 (+ 2711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:02:21.377264 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:02:21.377280 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:02:21.378165 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:02:21.379015 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:21.379043 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:21.380290 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:21.380299 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:21.382529 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:21.382548 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:21.383417 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:21.383425 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:21.383782 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:21.384858 (+ 1076us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:02:21.384894 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:02:21.388209 (+ 3315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:02:21.391537 (+ 3328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:02:21.391550 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:02:21.391571 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:02:21.392332 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:02:21.392739 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:02:21.392747 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:02:21.392753 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:02:21.393031 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:02:21.393604 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:02:21.393615 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:02:21.394158 (+ 543us) client_negotiation.cc:769] Sending connection context
0812 01:02:21.394536 (+ 378us) client_negotiation.cc:240] Negotiation successful
0812 01:02:21.395172 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":367,"threads_started":1}
I20260812 01:02:21.399398 17621 heartbeater.cc:347] Connected to a master server at 127.21.201.126:32907
I20260812 01:02:21.400003 17621 heartbeater.cc:464] Registering TS with master...
I20260812 01:02:21.400837 17621 heartbeater.cc:511] Master 127.21.201.126:32907 requested a full tablet report, sending...
I20260812 01:02:21.403178 17288 ts_manager.cc:194] Registered new tserver with Master: 1c6362753ad54842abda04607ea7cbd5 (127.21.201.65:41747)
I20260812 01:02:21.404798 17288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.65@KRBTEST.COM'} at 127.21.201.65:35795
WARNING: no policy specified for kudu/127.21.201.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
I20260812 01:02:21.445641 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:0
--local_ip_for_outbound_sockets=127.21.201.66
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:32907
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:39133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:02:21.798866 17628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:02:21.799151 17628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:02:21.799256 17628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:02:21.806753 17628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:02:21.806938 17628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:02:21.807147 17628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 01:02:21.817436 17628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:39133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:32907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:02:21.818751 17628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:02:21.820187 17628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:02:21.833676 17638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:02:21.833727 17640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:02:21.833758 17637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:02:21.834959 17628 server_base.cc:1034] running on GCE node
I20260812 01:02:21.835647 17628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:02:21.836684 17628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:02:21.837936 17628 hybrid_clock.cc:648] HybridClock initialized: now 1786496541837880 us; error 56 us; skew 500 ppm
I20260812 01:02:21.838282 17628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:02:21 dist-test-slave-7f01 krb5kdc[17209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496541, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:02:21.841624 17628 init.cc:377] Logged in from keytab as kudu/127.21.201.66@KRBTEST.COM (short username kudu)
I20260812 01:02:21.843235 17628 webserver.cc:460] Webserver started at http://127.21.201.66:37485/ using document root <none> and password file <none>
I20260812 01:02:21.844034 17628 fs_manager.cc:359] Metadata directory not provided
I20260812 01:02:21.844187 17628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:02:21.844534 17628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:02:21.847541 17628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/data/instance:
uuid: "cba9c8af143444528b5a7bc2c69e09a2"
format_stamp: "Formatted at 2026-08-12 01:02:21 on dist-test-slave-7f01"
server_key: "8dca22d8ec9ab7c87093148453b1ff41"
server_key_iv: "418a0bbe53e571e9fe241513faa6aacb"
server_key_version: "encryptionkey@0"
I20260812 01:02:21.848395 17628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance:
uuid: "cba9c8af143444528b5a7bc2c69e09a2"
format_stamp: "Formatted at 2026-08-12 01:02:21 on dist-test-slave-7f01"
server_key: "8dca22d8ec9ab7c87093148453b1ff41"
server_key_iv: "418a0bbe53e571e9fe241513faa6aacb"
server_key_version: "encryptionkey@0"
I20260812 01:02:21.853840 17628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:02:21.857497 17654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:21.858783 17628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260812 01:02:21.858964 17628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "cba9c8af143444528b5a7bc2c69e09a2"
format_stamp: "Formatted at 2026-08-12 01:02:21 on dist-test-slave-7f01"
server_key: "8dca22d8ec9ab7c87093148453b1ff41"
server_key_iv: "418a0bbe53e571e9fe241513faa6aacb"
server_key_version: "encryptionkey@0"
I20260812 01:02:21.859165 17628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:02:21.872041 17628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:02:21.876684 17628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:02:21.877017 17628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:02:21.878386 17628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:02:21.880787 17628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:02:21.880927 17628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:21.881078 17628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:02:21.881163 17628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:21.917273 17628 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:40855
I20260812 01:02:21.917366 17835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:40855 every 8 connection(s)
I20260812 01:02:21.918608 17628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 01:02:21.925985 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 17628
I20260812 01:02:21.926206 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance
I20260812 01:02:21.926692 22309 external_mini_cluster.cc:1428] Setting key a7e008f2c6b09de25ab93eae799bd56b
Aug 12 01:02:21 dist-test-slave-7f01 krb5kdc[17209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496541, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:02:21.941115 17839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:21.921866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:49367 (local address 127.21.201.126:32907)
0812 01:02:21.922433 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:21.922441 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:21.923469 (+ 1028us) server_negotiation.cc:407] Connection header received
0812 01:02:21.925630 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:21.925637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:21.925730 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:21.925882 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:02:21.927851 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:21.928839 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:21.930385 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:21.930847 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:21.930978 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:21.934948 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:02:21.934958 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:02:21.934962 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:02:21.934997 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:02:21.938070 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:21.938809 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:21.938814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:21.938816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:21.938962 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:21.939405 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:21.939411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:21.939413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:21.939668 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:02:21.939843 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:21.940496 (+ 653us) server_negotiation.cc:299] Negotiation successful
0812 01:02:21.940716 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":294,"threads_started":1}
I20260812 01:02:21.942044 17841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:21.922565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.66:49367)
0812 01:02:21.923307 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:21.923346 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 01:02:21.925352 (+ 2006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:02:21.926136 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:02:21.926148 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:02:21.926742 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:02:21.927634 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:21.927662 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:21.928996 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:21.929001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:21.930232 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:21.930245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:21.931082 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:21.931090 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:21.931340 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:21.932089 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:02:21.932111 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:02:21.934591 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:02:21.938214 (+ 3623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:02:21.938223 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:02:21.938238 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:02:21.938677 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:02:21.939073 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:02:21.939078 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:02:21.939082 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:02:21.939272 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:02:21.939815 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:02:21.939823 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:02:21.940147 (+ 324us) client_negotiation.cc:769] Sending connection context
0812 01:02:21.940398 (+ 251us) client_negotiation.cc:240] Negotiation successful
0812 01:02:21.940911 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":294,"threads_started":1}
I20260812 01:02:21.943552 17836 heartbeater.cc:347] Connected to a master server at 127.21.201.126:32907
I20260812 01:02:21.943893 17836 heartbeater.cc:464] Registering TS with master...
I20260812 01:02:21.944613 17836 heartbeater.cc:511] Master 127.21.201.126:32907 requested a full tablet report, sending...
I20260812 01:02:21.946065 17288 ts_manager.cc:194] Registered new tserver with Master: cba9c8af143444528b5a7bc2c69e09a2 (127.21.201.66:40855)
I20260812 01:02:21.946892 17288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.66@KRBTEST.COM'} at 127.21.201.66:49367
WARNING: no policy specified for kudu/127.21.201.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
I20260812 01:02:21.994870 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:0
--local_ip_for_outbound_sockets=127.21.201.67
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:32907
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:39133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:02:22.368467 17847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:02:22.368755 17847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:02:22.368927 17847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:02:22.376735 17847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:02:22.376852 17847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:02:22.377031 17847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 01:02:22.387171 17847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:39133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:32907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:02:22.388587 17847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:02:22.390069 17847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:02:22.406226 17853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:02:22.406275 17855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:02:22.406201 17857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:02:22.407651 17847 server_base.cc:1034] running on GCE node
I20260812 01:02:22.408494 17847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:02:22.408946 17621 heartbeater.cc:503] Master 127.21.201.126:32907 was elected leader, sending a full tablet report...
I20260812 01:02:22.409842 17847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:02:22.411154 17847 hybrid_clock.cc:648] HybridClock initialized: now 1786496542411093 us; error 51 us; skew 500 ppm
I20260812 01:02:22.411506 17847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:02:22 dist-test-slave-7f01 krb5kdc[17209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496542, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:02:22.415102 17847 init.cc:377] Logged in from keytab as kudu/127.21.201.67@KRBTEST.COM (short username kudu)
I20260812 01:02:22.416777 17847 webserver.cc:460] Webserver started at http://127.21.201.67:34809/ using document root <none> and password file <none>
I20260812 01:02:22.417529 17847 fs_manager.cc:359] Metadata directory not provided
I20260812 01:02:22.417687 17847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:02:22.418016 17847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:02:22.421468 17847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/data/instance:
uuid: "762b216bab804e69ae171e5ae21e1bd1"
format_stamp: "Formatted at 2026-08-12 01:02:22 on dist-test-slave-7f01"
server_key: "508e56cdcb206f2d1a87853429c21f0b"
server_key_iv: "92f0e3a670001f51d1d3acf730a87bd0"
server_key_version: "encryptionkey@0"
I20260812 01:02:22.422305 17847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance:
uuid: "762b216bab804e69ae171e5ae21e1bd1"
format_stamp: "Formatted at 2026-08-12 01:02:22 on dist-test-slave-7f01"
server_key: "508e56cdcb206f2d1a87853429c21f0b"
server_key_iv: "92f0e3a670001f51d1d3acf730a87bd0"
server_key_version: "encryptionkey@0"
I20260812 01:02:22.428418 17847 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 01:02:22.432461 17870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:22.433967 17847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 01:02:22.434168 17847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "762b216bab804e69ae171e5ae21e1bd1"
format_stamp: "Formatted at 2026-08-12 01:02:22 on dist-test-slave-7f01"
server_key: "508e56cdcb206f2d1a87853429c21f0b"
server_key_iv: "92f0e3a670001f51d1d3acf730a87bd0"
server_key_version: "encryptionkey@0"
I20260812 01:02:22.434396 17847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:02:22.454684 17847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:02:22.459103 17847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:02:22.459436 17847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:02:22.460551 17847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:02:22.461990 17847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:02:22.462095 17847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:22.462250 17847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:02:22.462325 17847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:22.502533 17847 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:35975
I20260812 01:02:22.502583 18047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:35975 every 8 connection(s)
I20260812 01:02:22.503944 17847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 01:02:22.507853 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 17847
I20260812 01:02:22.508107 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance
I20260812 01:02:22.508809 22309 external_mini_cluster.cc:1428] Setting key 7aa47ce7e10a450730adaf1e03e83521
Aug 12 01:02:22 dist-test-slave-7f01 krb5kdc[17209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496542, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:02:22.529934 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:22.506715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:50861 (local address 127.21.201.126:32907)
0812 01:02:22.508025 (+ 1310us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:22.508033 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:22.508227 (+ 194us) server_negotiation.cc:407] Connection header received
0812 01:02:22.510549 (+ 2322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:22.510555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:22.510655 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:22.510763 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:02:22.512581 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.513433 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.515174 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.515614 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.515712 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:22.519855 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:02:22.519865 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:02:22.519870 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:02:22.519954 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:02:22.524596 (+ 4642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:22.526268 (+ 1672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:22.526273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:22.526275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:22.526428 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:22.527153 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:22.527158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:22.527161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:22.527836 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:02:22.528066 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:22.529388 (+ 1322us) server_negotiation.cc:299] Negotiation successful
0812 01:02:22.529609 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1143,"thread_start_us":575,"threads_started":1}
I20260812 01:02:22.530983 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:22.507115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.67:50861)
0812 01:02:22.508038 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:22.508099 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 01:02:22.510246 (+ 2147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:02:22.511080 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:02:22.511089 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:02:22.511650 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:02:22.512417 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:22.512435 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.513589 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.513593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:22.515018 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:22.515031 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.515899 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.515905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:22.516116 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:22.516776 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:02:22.516804 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:02:22.519363 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:02:22.525277 (+ 5914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:02:22.525290 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:02:22.525311 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:02:22.526015 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:02:22.526586 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:02:22.526592 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:02:22.526594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:02:22.526876 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:02:22.528104 (+ 1228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:02:22.528114 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:02:22.528884 (+ 770us) client_negotiation.cc:769] Sending connection context
0812 01:02:22.529205 (+ 321us) client_negotiation.cc:240] Negotiation successful
0812 01:02:22.529690 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":378,"threads_started":1}
I20260812 01:02:22.532655 18048 heartbeater.cc:347] Connected to a master server at 127.21.201.126:32907
I20260812 01:02:22.532994 18048 heartbeater.cc:464] Registering TS with master...
I20260812 01:02:22.533799 18048 heartbeater.cc:511] Master 127.21.201.126:32907 requested a full tablet report, sending...
I20260812 01:02:22.535609 17288 ts_manager.cc:194] Registered new tserver with Master: 762b216bab804e69ae171e5ae21e1bd1 (127.21.201.67:35975)
I20260812 01:02:22.536975 17288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.67@KRBTEST.COM'} at 127.21.201.67:50861
I20260812 01:02:22.538730 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:02:22.557268 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:22.543871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51436 (local address 127.21.201.126:32907)
0812 01:02:22.544053 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:22.544058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:22.544666 (+ 608us) server_negotiation.cc:407] Connection header received
0812 01:02:22.544815 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:22.544818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:22.544884 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:22.544997 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:02:22.546480 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.547469 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.548958 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.549369 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.549483 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:22.552075 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:02:22.552080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:02:22.552083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:02:22.552117 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:02:22.554720 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:22.555414 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:22.555420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:22.555424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:22.555562 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:22.555987 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:22.555991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:22.555993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:22.556350 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:02:22.556560 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:22.556807 (+ 247us) server_negotiation.cc:299] Negotiation successful
0812 01:02:22.557016 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 01:02:22.561460 17288 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:02:22.568902 17288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:02:22.594496 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:22.582159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:41747 (local address 127.0.0.1:36230)
0812 01:02:22.584481 (+ 2322us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:22.584500 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:02:22.584629 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:02:22.585585 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:02:22.585593 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:02:22.585635 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:02:22.586191 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:22.586205 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.588379 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.588384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:22.591020 (+ 2636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:22.591036 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.593408 (+ 2372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.593416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:22.593647 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:22.593707 (+ 60us) client_negotiation.cc:769] Sending connection context
0812 01:02:22.593862 (+ 155us) client_negotiation.cc:240] Negotiation successful
0812 01:02:22.594074 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2205,"thread_start_us":528,"threads_started":1}
I20260812 01:02:22.595322 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:22.583064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36230 (local address 127.21.201.65:41747)
0812 01:02:22.584138 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:22.584150 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:22.584750 (+ 600us) server_negotiation.cc:407] Connection header received
0812 01:02:22.584870 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:22.584878 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:22.585094 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:22.585291 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:02:22.586418 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.588206 (+ 1788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.591230 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.592860 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.593004 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:22.593271 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:22.594777 (+ 1506us) server_negotiation.cc:299] Negotiation successful
0812 01:02:22.594976 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822,"thread_start_us":652,"threads_started":1}
I20260812 01:02:22.596625 18069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:22.581599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:35975 (local address 127.0.0.1:38542)
0812 01:02:22.583133 (+ 1534us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:22.583177 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 01:02:22.583427 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:02:22.585972 (+ 2545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:02:22.585977 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:02:22.586012 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:02:22.586485 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:22.586502 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.589970 (+ 3468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.589975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:22.592189 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:22.592205 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.595761 (+ 3556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.595769 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:22.595980 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:22.596012 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 01:02:22.596083 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 01:02:22.596235 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1281,"thread_start_us":2612,"threads_started":1}
I20260812 01:02:22.597981 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:22.582554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38542 (local address 127.21.201.67:35975)
0812 01:02:22.584094 (+ 1540us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:22.584106 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:22.584151 (+ 45us) server_negotiation.cc:407] Connection header received
0812 01:02:22.584268 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:22.584278 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:22.584564 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:22.586699 (+ 2135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:02:22.587011 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.589814 (+ 2803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.592426 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.595367 (+ 2941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.597047 (+ 1680us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:22.597304 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:22.597385 (+ 81us) server_negotiation.cc:299] Negotiation successful
0812 01:02:22.597577 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1350,"thread_start_us":1356,"threads_started":1}
I20260812 01:02:22.600517 17514 tablet_service.cc:1467] Processing CreateTablet for tablet 93ba78295859407da3866297f1080f35 (DEFAULT_TABLE table=test-table [id=1e6322fcea6d47ffbc671b28dbaa0214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:02:22.602155 17514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93ba78295859407da3866297f1080f35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:02:22.601924 17948 tablet_service.cc:1467] Processing CreateTablet for tablet 93ba78295859407da3866297f1080f35 (DEFAULT_TABLE table=test-table [id=1e6322fcea6d47ffbc671b28dbaa0214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:02:22.603540 17948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93ba78295859407da3866297f1080f35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:02:22.605640 18076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:22.584891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52686 (local address 127.21.201.66:40855)
0812 01:02:22.597239 (+ 12348us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:22.597249 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:22.597282 (+ 33us) server_negotiation.cc:407] Connection header received
0812 01:02:22.597400 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:22.597408 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:22.597689 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:22.597923 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:02:22.598632 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.600448 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.602788 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.604414 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.604579 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:22.604786 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:22.605056 (+ 270us) server_negotiation.cc:299] Negotiation successful
0812 01:02:22.605247 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12170,"thread_start_us":12043,"threads_started":1}
I20260812 01:02:22.606511 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:22.584590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:40855 (local address 127.0.0.1:52686)
0812 01:02:22.596020 (+ 11430us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:22.596040 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:02:22.596136 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:02:22.598221 (+ 2085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:02:22.598225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:02:22.598257 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:02:22.598438 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:22.598448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.600635 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.600639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:22.602573 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:22.602590 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.604627 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.604634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:22.604834 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:22.604875 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 01:02:22.604981 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 01:02:22.606157 (+ 1176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11326,"thread_start_us":11970,"threads_started":1}
I20260812 01:02:22.610780 17722 tablet_service.cc:1467] Processing CreateTablet for tablet 93ba78295859407da3866297f1080f35 (DEFAULT_TABLE table=test-table [id=1e6322fcea6d47ffbc671b28dbaa0214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:02:22.612310 17722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93ba78295859407da3866297f1080f35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:02:22.615780 18079 tablet_bootstrap.cc:492] T 93ba78295859407da3866297f1080f35 P 1c6362753ad54842abda04607ea7cbd5: Bootstrap starting.
I20260812 01:02:22.618425 18080 tablet_bootstrap.cc:492] T 93ba78295859407da3866297f1080f35 P 762b216bab804e69ae171e5ae21e1bd1: Bootstrap starting.
I20260812 01:02:22.619597 18079 tablet_bootstrap.cc:654] T 93ba78295859407da3866297f1080f35 P 1c6362753ad54842abda04607ea7cbd5: Neither blocks nor log segments found. Creating new log.
I20260812 01:02:22.621112 18079 log.cc:824] T 93ba78295859407da3866297f1080f35 P 1c6362753ad54842abda04607ea7cbd5: Log is configured to *not* fsync() on all Append() calls
I20260812 01:02:22.622361 18080 tablet_bootstrap.cc:654] T 93ba78295859407da3866297f1080f35 P 762b216bab804e69ae171e5ae21e1bd1: Neither blocks nor log segments found. Creating new log.
I20260812 01:02:22.624084 18080 log.cc:824] T 93ba78295859407da3866297f1080f35 P 762b216bab804e69ae171e5ae21e1bd1: Log is configured to *not* fsync() on all Append() calls
I20260812 01:02:22.626837 18084 tablet_bootstrap.cc:492] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2: Bootstrap starting.
I20260812 01:02:22.627120 18079 tablet_bootstrap.cc:492] T 93ba78295859407da3866297f1080f35 P 1c6362753ad54842abda04607ea7cbd5: No bootstrap required, opened a new log
I20260812 01:02:22.627506 18079 ts_tablet_manager.cc:1397] T 93ba78295859407da3866297f1080f35 P 1c6362753ad54842abda04607ea7cbd5: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260812 01:02:22.630481 18080 tablet_bootstrap.cc:492] T 93ba78295859407da3866297f1080f35 P 762b216bab804e69ae171e5ae21e1bd1: No bootstrap required, opened a new log
I20260812 01:02:22.630779 18080 ts_tablet_manager.cc:1397] T 93ba78295859407da3866297f1080f35 P 762b216bab804e69ae171e5ae21e1bd1: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260812 01:02:22.630802 18084 tablet_bootstrap.cc:654] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2: Neither blocks nor log segments found. Creating new log.
I20260812 01:02:22.631930 18084 log.cc:824] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2: Log is configured to *not* fsync() on all Append() calls
I20260812 01:02:22.638536 18079 raft_consensus.cc:348] T 93ba78295859407da3866297f1080f35 P 1c6362753ad54842abda04607ea7cbd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762b216bab804e69ae171e5ae21e1bd1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35975 } } peers { permanent_uuid: "1c6362753ad54842abda04607ea7cbd5" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 41747 } } peers { permanent_uuid: "cba9c8af143444528b5a7bc2c69e09a2" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40855 } }
I20260812 01:02:22.638955 18084 tablet_bootstrap.cc:492] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2: No bootstrap required, opened a new log
I20260812 01:02:22.639179 18079 raft_consensus.cc:374] T 93ba78295859407da3866297f1080f35 P 1c6362753ad54842abda04607ea7cbd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:02:22.639341 18079 raft_consensus.cc:729] T 93ba78295859407da3866297f1080f35 P 1c6362753ad54842abda04607ea7cbd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c6362753ad54842abda04607ea7cbd5, State: Initialized, Role: FOLLOWER
I20260812 01:02:22.639479 18084 ts_tablet_manager.cc:1397] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260812 01:02:22.640185 18079 consensus_queue.cc:260] T 93ba78295859407da3866297f1080f35 P 1c6362753ad54842abda04607ea7cbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762b216bab804e69ae171e5ae21e1bd1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35975 } } peers { permanent_uuid: "1c6362753ad54842abda04607ea7cbd5" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 41747 } } peers { permanent_uuid: "cba9c8af143444528b5a7bc2c69e09a2" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40855 } }
I20260812 01:02:22.642153 18079 ts_tablet_manager.cc:1428] T 93ba78295859407da3866297f1080f35 P 1c6362753ad54842abda04607ea7cbd5: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260812 01:02:22.642102 18080 raft_consensus.cc:348] T 93ba78295859407da3866297f1080f35 P 762b216bab804e69ae171e5ae21e1bd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762b216bab804e69ae171e5ae21e1bd1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35975 } } peers { permanent_uuid: "1c6362753ad54842abda04607ea7cbd5" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 41747 } } peers { permanent_uuid: "cba9c8af143444528b5a7bc2c69e09a2" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40855 } }
I20260812 01:02:22.642755 18080 raft_consensus.cc:374] T 93ba78295859407da3866297f1080f35 P 762b216bab804e69ae171e5ae21e1bd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:02:22.642882 18080 raft_consensus.cc:729] T 93ba78295859407da3866297f1080f35 P 762b216bab804e69ae171e5ae21e1bd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 762b216bab804e69ae171e5ae21e1bd1, State: Initialized, Role: FOLLOWER
I20260812 01:02:22.643527 18080 consensus_queue.cc:260] T 93ba78295859407da3866297f1080f35 P 762b216bab804e69ae171e5ae21e1bd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762b216bab804e69ae171e5ae21e1bd1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35975 } } peers { permanent_uuid: "1c6362753ad54842abda04607ea7cbd5" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 41747 } } peers { permanent_uuid: "cba9c8af143444528b5a7bc2c69e09a2" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40855 } }
I20260812 01:02:22.644737 18048 heartbeater.cc:503] Master 127.21.201.126:32907 was elected leader, sending a full tablet report...
I20260812 01:02:22.645349 18080 ts_tablet_manager.cc:1428] T 93ba78295859407da3866297f1080f35 P 762b216bab804e69ae171e5ae21e1bd1: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260812 01:02:22.651481 18084 raft_consensus.cc:348] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762b216bab804e69ae171e5ae21e1bd1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35975 } } peers { permanent_uuid: "1c6362753ad54842abda04607ea7cbd5" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 41747 } } peers { permanent_uuid: "cba9c8af143444528b5a7bc2c69e09a2" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40855 } }
I20260812 01:02:22.652027 18084 raft_consensus.cc:374] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:02:22.652143 18084 raft_consensus.cc:729] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cba9c8af143444528b5a7bc2c69e09a2, State: Initialized, Role: FOLLOWER
I20260812 01:02:22.652727 18084 consensus_queue.cc:260] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762b216bab804e69ae171e5ae21e1bd1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35975 } } peers { permanent_uuid: "1c6362753ad54842abda04607ea7cbd5" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 41747 } } peers { permanent_uuid: "cba9c8af143444528b5a7bc2c69e09a2" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40855 } }
I20260812 01:02:22.654210 17836 heartbeater.cc:503] Master 127.21.201.126:32907 was elected leader, sending a full tablet report...
I20260812 01:02:22.654527 18084 ts_tablet_manager.cc:1428] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260812 01:02:22.671156 17838 tablet.cc:2369] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:02:22.755851 18051 tablet.cc:2369] T 93ba78295859407da3866297f1080f35 P 762b216bab804e69ae171e5ae21e1bd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:02:22.849784 18092 raft_consensus.cc:482] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:02:22.850163 18092 raft_consensus.cc:504] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762b216bab804e69ae171e5ae21e1bd1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35975 } } peers { permanent_uuid: "1c6362753ad54842abda04607ea7cbd5" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 41747 } } peers { permanent_uuid: "cba9c8af143444528b5a7bc2c69e09a2" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40855 } }
I20260812 01:02:22.852048 18092 leader_election.cc:288] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 762b216bab804e69ae171e5ae21e1bd1 (127.21.201.67:35975), 1c6362753ad54842abda04607ea7cbd5 (127.21.201.65:41747)
I20260812 01:02:22.859072 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:22.853060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:43417 (local address 127.21.201.65:41747)
0812 01:02:22.853534 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:22.853542 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:22.853566 (+ 24us) server_negotiation.cc:407] Connection header received
0812 01:02:22.853658 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:22.853662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:22.853721 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:22.853827 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:02:22.854418 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.855191 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.857246 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.858057 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.858271 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:22.858364 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:22.858636 (+ 272us) server_negotiation.cc:299] Negotiation successful
0812 01:02:22.858768 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318}
I20260812 01:02:22.859752 18093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:22.852425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:41747 (local address 127.21.201.66:43417)
0812 01:02:22.853265 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:22.853288 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 01:02:22.853484 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:02:22.853855 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:02:22.853860 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:02:22.853887 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:02:22.854253 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:22.854262 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.855329 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.855336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:22.857075 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:22.857088 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.858294 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.858300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:22.858460 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:22.858484 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 01:02:22.858592 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 01:02:22.858741 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":1,"thread_start_us":418,"threads_started":1}
I20260812 01:02:22.859899 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:22.853284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:53919 (local address 127.21.201.67:35975)
0812 01:02:22.853655 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:22.853663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:22.853688 (+ 25us) server_negotiation.cc:407] Connection header received
0812 01:02:22.853937 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:22.853941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:22.854005 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:22.854103 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:02:22.854720 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.855453 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.857639 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.858490 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.858704 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:22.858779 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:22.859178 (+ 399us) server_negotiation.cc:299] Negotiation successful
0812 01:02:22.859450 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221}
I20260812 01:02:22.860378 18097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:22.852538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:35975 (local address 127.21.201.66:53919)
0812 01:02:22.853604 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:22.853629 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 01:02:22.853768 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:02:22.854137 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:02:22.854141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:02:22.854168 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:02:22.854561 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:22.854569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.855607 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.855613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:22.857458 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:22.857472 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.858732 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.858739 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:22.858940 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:22.858983 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 01:02:22.859093 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 01:02:22.859226 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"mutex_wait_us":23,"thread_start_us":724,"threads_started":1}
I20260812 01:02:22.861032 17552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93ba78295859407da3866297f1080f35" candidate_uuid: "cba9c8af143444528b5a7bc2c69e09a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c6362753ad54842abda04607ea7cbd5" is_pre_election: true
I20260812 01:02:22.861387 17979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93ba78295859407da3866297f1080f35" candidate_uuid: "cba9c8af143444528b5a7bc2c69e09a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762b216bab804e69ae171e5ae21e1bd1" is_pre_election: true
I20260812 01:02:22.861599 17552 raft_consensus.cc:2454] T 93ba78295859407da3866297f1080f35 P 1c6362753ad54842abda04607ea7cbd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cba9c8af143444528b5a7bc2c69e09a2 in term 0.
I20260812 01:02:22.861905 17979 raft_consensus.cc:2454] T 93ba78295859407da3866297f1080f35 P 762b216bab804e69ae171e5ae21e1bd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cba9c8af143444528b5a7bc2c69e09a2 in term 0.
I20260812 01:02:22.862769 17660 leader_election.cc:302] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c6362753ad54842abda04607ea7cbd5, cba9c8af143444528b5a7bc2c69e09a2; no voters:
I20260812 01:02:22.863252 18092 raft_consensus.cc:2781] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:02:22.863421 18092 raft_consensus.cc:482] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:02:22.863552 18092 raft_consensus.cc:3037] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:02:22.866041 18092 raft_consensus.cc:504] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762b216bab804e69ae171e5ae21e1bd1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35975 } } peers { permanent_uuid: "1c6362753ad54842abda04607ea7cbd5" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 41747 } } peers { permanent_uuid: "cba9c8af143444528b5a7bc2c69e09a2" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40855 } }
I20260812 01:02:22.866719 18092 leader_election.cc:288] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2 [CANDIDATE]: Term 1 election: Requested vote from peers 762b216bab804e69ae171e5ae21e1bd1 (127.21.201.67:35975), 1c6362753ad54842abda04607ea7cbd5 (127.21.201.65:41747)
I20260812 01:02:22.867288 17552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93ba78295859407da3866297f1080f35" candidate_uuid: "cba9c8af143444528b5a7bc2c69e09a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c6362753ad54842abda04607ea7cbd5"
I20260812 01:02:22.867522 17552 raft_consensus.cc:3037] T 93ba78295859407da3866297f1080f35 P 1c6362753ad54842abda04607ea7cbd5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:02:22.867609 17979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93ba78295859407da3866297f1080f35" candidate_uuid: "cba9c8af143444528b5a7bc2c69e09a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762b216bab804e69ae171e5ae21e1bd1"
I20260812 01:02:22.868024 17979 raft_consensus.cc:3037] T 93ba78295859407da3866297f1080f35 P 762b216bab804e69ae171e5ae21e1bd1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:02:22.870009 17552 raft_consensus.cc:2454] T 93ba78295859407da3866297f1080f35 P 1c6362753ad54842abda04607ea7cbd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cba9c8af143444528b5a7bc2c69e09a2 in term 1.
I20260812 01:02:22.870494 17660 leader_election.cc:302] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c6362753ad54842abda04607ea7cbd5, cba9c8af143444528b5a7bc2c69e09a2; no voters:
I20260812 01:02:22.870823 18092 raft_consensus.cc:2781] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2 [term 1 FOLLOWER]: Leader election won for term 1
W20260812 01:02:22.870891 17622 tablet.cc:2369] T 93ba78295859407da3866297f1080f35 P 1c6362753ad54842abda04607ea7cbd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:02:22.871292 17979 raft_consensus.cc:2454] T 93ba78295859407da3866297f1080f35 P 762b216bab804e69ae171e5ae21e1bd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cba9c8af143444528b5a7bc2c69e09a2 in term 1.
I20260812 01:02:22.871968 18092 raft_consensus.cc:686] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2 [term 1 LEADER]: Becoming Leader. State: Replica: cba9c8af143444528b5a7bc2c69e09a2, State: Running, Role: LEADER
I20260812 01:02:22.872490 18092 consensus_queue.cc:237] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762b216bab804e69ae171e5ae21e1bd1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35975 } } peers { permanent_uuid: "1c6362753ad54842abda04607ea7cbd5" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 41747 } } peers { permanent_uuid: "cba9c8af143444528b5a7bc2c69e09a2" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40855 } }
I20260812 01:02:22.877368 17286 catalog_manager.cc:5495] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to cba9c8af143444528b5a7bc2c69e09a2 (127.21.201.66). New cstate: current_term: 1 leader_uuid: "cba9c8af143444528b5a7bc2c69e09a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762b216bab804e69ae171e5ae21e1bd1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c6362753ad54842abda04607ea7cbd5" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 41747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cba9c8af143444528b5a7bc2c69e09a2" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40855 } health_report { overall_health: HEALTHY } } }
I20260812 01:02:22.966576 18076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:22.960385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52694 (local address 127.21.201.66:40855)
0812 01:02:22.960675 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:22.960681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:22.960702 (+ 21us) server_negotiation.cc:407] Connection header received
0812 01:02:22.960851 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:22.960856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:22.960939 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:22.961116 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:02:22.961518 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.962259 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.964212 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:22.964638 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:22.964740 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:22.965252 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:02:22.965845 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:02:22.965961 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:22.966164 (+ 203us) server_negotiation.cc:299] Negotiation successful
0812 01:02:22.966237 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260812 01:02:22.975237 17552 raft_consensus.cc:1261] T 93ba78295859407da3866297f1080f35 P 1c6362753ad54842abda04607ea7cbd5 [term 1 FOLLOWER]: Refusing update from remote peer cba9c8af143444528b5a7bc2c69e09a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:02:22.975301 17979 raft_consensus.cc:1261] T 93ba78295859407da3866297f1080f35 P 762b216bab804e69ae171e5ae21e1bd1 [term 1 FOLLOWER]: Refusing update from remote peer cba9c8af143444528b5a7bc2c69e09a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:02:22.976301 18099 consensus_queue.cc:1035] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "762b216bab804e69ae171e5ae21e1bd1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:02:22.976733 18092 consensus_queue.cc:1035] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c6362753ad54842abda04607ea7cbd5" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 41747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:02:23.002879 17286 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:02:23.003167 17286 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:02:23.007258 17286 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3: Sending DeleteTablet for 3 replicas of tablet 93ba78295859407da3866297f1080f35
I20260812 01:02:23.008616 17514 tablet_service.cc:1514] Processing DeleteTablet for tablet 93ba78295859407da3866297f1080f35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:02:22 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:36230
I20260812 01:02:23.008616 17948 tablet_service.cc:1514] Processing DeleteTablet for tablet 93ba78295859407da3866297f1080f35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:02:22 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:38542
I20260812 01:02:23.008891 17722 tablet_service.cc:1514] Processing DeleteTablet for tablet 93ba78295859407da3866297f1080f35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:02:22 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:52686
I20260812 01:02:23.010213 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 17394
I20260812 01:02:23.019546 18123 tablet_replica.cc:330] T 93ba78295859407da3866297f1080f35 P 762b216bab804e69ae171e5ae21e1bd1: stopping tablet replica
I20260812 01:02:23.020273 18123 raft_consensus.cc:2229] T 93ba78295859407da3866297f1080f35 P 762b216bab804e69ae171e5ae21e1bd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:02:23.021008 18123 raft_consensus.cc:2258] T 93ba78295859407da3866297f1080f35 P 762b216bab804e69ae171e5ae21e1bd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:02:23.021696 18125 tablet_replica.cc:330] T 93ba78295859407da3866297f1080f35 P 1c6362753ad54842abda04607ea7cbd5: stopping tablet replica
I20260812 01:02:23.022810 18125 raft_consensus.cc:2229] T 93ba78295859407da3866297f1080f35 P 1c6362753ad54842abda04607ea7cbd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:02:23.025352 18123 ts_tablet_manager.cc:1905] T 93ba78295859407da3866297f1080f35 P 762b216bab804e69ae171e5ae21e1bd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:02:23.025964 18125 raft_consensus.cc:2258] T 93ba78295859407da3866297f1080f35 P 1c6362753ad54842abda04607ea7cbd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:02:23.027148 18126 tablet_replica.cc:330] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2: stopping tablet replica
I20260812 01:02:23.027556 18126 raft_consensus.cc:2229] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2 [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:02:23.028333 18126 raft_consensus.cc:2258] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:02:23.030422 18126 ts_tablet_manager.cc:1905] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:02:23.034490 18123 ts_tablet_manager.cc:1918] T 93ba78295859407da3866297f1080f35 P 762b216bab804e69ae171e5ae21e1bd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:02:23.034725 18123 log.cc:1196] T 93ba78295859407da3866297f1080f35 P 762b216bab804e69ae171e5ae21e1bd1: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/93ba78295859407da3866297f1080f35
I20260812 01:02:23.034793 18125 ts_tablet_manager.cc:1905] T 93ba78295859407da3866297f1080f35 P 1c6362753ad54842abda04607ea7cbd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:02:23.035430 18123 ts_tablet_manager.cc:1939] T 93ba78295859407da3866297f1080f35 P 762b216bab804e69ae171e5ae21e1bd1: Deleting consensus metadata
I20260812 01:02:23.037698 17263 catalog_manager.cc:4841] TS 762b216bab804e69ae171e5ae21e1bd1 (127.21.201.67:35975): tablet 93ba78295859407da3866297f1080f35 (table test-table [id=1e6322fcea6d47ffbc671b28dbaa0214]) successfully deleted
I20260812 01:02:23.038352 18126 ts_tablet_manager.cc:1918] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:02:23.038578 18126 log.cc:1196] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/93ba78295859407da3866297f1080f35
I20260812 01:02:23.039170 18126 ts_tablet_manager.cc:1939] T 93ba78295859407da3866297f1080f35 P cba9c8af143444528b5a7bc2c69e09a2: Deleting consensus metadata
I20260812 01:02:23.040889 17263 catalog_manager.cc:4841] TS cba9c8af143444528b5a7bc2c69e09a2 (127.21.201.66:40855): tablet 93ba78295859407da3866297f1080f35 (table test-table [id=1e6322fcea6d47ffbc671b28dbaa0214]) successfully deleted
I20260812 01:02:23.042541 18125 ts_tablet_manager.cc:1918] T 93ba78295859407da3866297f1080f35 P 1c6362753ad54842abda04607ea7cbd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:02:23.042735 18125 log.cc:1196] T 93ba78295859407da3866297f1080f35 P 1c6362753ad54842abda04607ea7cbd5: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/wal/wals/93ba78295859407da3866297f1080f35
I20260812 01:02:23.043445 18125 ts_tablet_manager.cc:1939] T 93ba78295859407da3866297f1080f35 P 1c6362753ad54842abda04607ea7cbd5: Deleting consensus metadata
I20260812 01:02:23.045581 17264 catalog_manager.cc:4841] TS 1c6362753ad54842abda04607ea7cbd5 (127.21.201.65:41747): tablet 93ba78295859407da3866297f1080f35 (table test-table [id=1e6322fcea6d47ffbc671b28dbaa0214]) successfully deleted
Aug 12 01:02:23 dist-test-slave-7f01 krb5kdc[17209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.65@KRBTEST.COM
I20260812 01:02:23.056928 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:23.011664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36238 (local address 127.21.201.65:41747)
0812 01:02:23.011929 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:23.011939 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:23.026227 (+ 14288us) server_negotiation.cc:407] Connection header received
0812 01:02:23.026649 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:23.026657 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:23.026797 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:23.026918 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:02:23.029193 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:23.030306 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:23.032926 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:23.033594 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:23.033698 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:23.048939 (+ 15241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:02:23.048959 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:02:23.048980 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:02:23.049049 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:02:23.053465 (+ 4416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:23.054401 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:23.054421 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:23.054425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:23.054604 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:23.055071 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:23.055077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:23.055081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:23.055585 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:02:23.055940 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:23.056259 (+ 319us) server_negotiation.cc:299] Negotiation successful
0812 01:02:23.056519 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 01:02:23.057735 17614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:02:23.152339 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 17394
I20260812 01:02:23.179436 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 17628
Aug 12 01:02:23 dist-test-slave-7f01 krb5kdc[17209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.66@KRBTEST.COM
I20260812 01:02:23.197507 18076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:23.180460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52710 (local address 127.21.201.66:40855)
0812 01:02:23.180665 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:23.180672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:23.180693 (+ 21us) server_negotiation.cc:407] Connection header received
0812 01:02:23.180774 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:23.180779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:23.180873 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:23.181003 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:02:23.182259 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:23.183540 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:23.184927 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:23.185613 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:23.185740 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:23.190504 (+ 4764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:02:23.190520 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:02:23.190540 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:02:23.190586 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:02:23.194576 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:23.195250 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:23.195259 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:23.195263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:23.195447 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:23.195854 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:23.195860 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:23.195863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:23.196311 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:02:23.196651 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:23.196875 (+ 224us) server_negotiation.cc:299] Negotiation successful
0812 01:02:23.197113 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 01:02:23.198326 17831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:02:23.317027 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 17628
I20260812 01:02:23.345718 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 17847
Aug 12 01:02:23 dist-test-slave-7f01 krb5kdc[17209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.67@KRBTEST.COM
I20260812 01:02:23.361481 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:23.346556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38546 (local address 127.21.201.67:35975)
0812 01:02:23.346752 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:23.346759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:23.346893 (+ 134us) server_negotiation.cc:407] Connection header received
0812 01:02:23.347144 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:23.347148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:23.347235 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:23.347361 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:02:23.348782 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:23.349541 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:23.350927 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:23.351327 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:23.351471 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:23.356118 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:02:23.356130 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:02:23.356145 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:02:23.356177 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:02:23.358811 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:23.359496 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:23.359503 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:23.359506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:23.359686 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:23.360134 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:23.360138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:23.360140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:23.360536 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:02:23.360771 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:23.361024 (+ 253us) server_negotiation.cc:299] Negotiation successful
0812 01:02:23.361194 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 01:02:23.362149 18043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:02:23.454449 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 17847
I20260812 01:02:23.480466 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 17232
I20260812 01:02:23.481650 17344 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:02:23.596908 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 17232
I20260812 01:02:43.629343 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:32907
--webserver_interface=127.21.201.126
--webserver_port=38777
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:39133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:32907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:02:44.012352 18133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:02:44.012679 18133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:02:44.012842 18133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:02:44.020856 18133 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 01:02:44.020980 18133 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 01:02:44.021051 18133 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 01:02:44.021153 18133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:02:44.021257 18133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:02:44.021364 18133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:02:44.021435 18133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:02:44.021515 18133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:02:44.031622 18133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:39133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:32907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:32907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=38777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:02:44.033013 18133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:02:44.034382 18133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:02:44.060981 18144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:02:44.060896 18150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:02:44.060871 18145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:02:44.061939 18133 server_base.cc:1034] running on GCE node
I20260812 01:02:44.062947 18133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:02:44.064682 18133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:02:44.065909 18133 hybrid_clock.cc:648] HybridClock initialized: now 1786496564065831 us; error 62 us; skew 500 ppm
I20260812 01:02:44.066335 18133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:02:44 dist-test-slave-7f01 krb5kdc[17209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496564, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:02:44.070334 18133 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 01:02:44.072263 18133 webserver.cc:460] Webserver started at http://127.21.201.126:38777/ using document root <none> and password file <none>
I20260812 01:02:44.073104 18133 fs_manager.cc:359] Metadata directory not provided
I20260812 01:02:44.073232 18133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:02:44.079519 18133 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 01:02:44.082518 18160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:44.084038 18133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 01:02:44.084226 18133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "1409835d19b84f5c84fef34e4cc8e8d3"
format_stamp: "Formatted at 2026-08-12 01:02:20 on dist-test-slave-7f01"
server_key: "e2b99432c1792435d918d9268787e6de"
server_key_iv: "25f718051a7f4176395a81a6be28962f"
server_key_version: "encryptionkey@0"
I20260812 01:02:44.085347 18133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:02:44.106078 18133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:02:44.110534 18133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:02:44.110900 18133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:02:44.132038 18133 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:32907
I20260812 01:02:44.132289 18248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:32907 every 8 connection(s)
I20260812 01:02:44.133944 18133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:02:44.134644 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 18133
I20260812 01:02:44.135805 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:41747
--local_ip_for_outbound_sockets=127.21.201.65
--tserver_master_addrs=127.21.201.126:32907
--webserver_port=33643
--webserver_interface=127.21.201.65
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:39133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:02:44.149166 18249 sys_catalog.cc:263] Verifying existing consensus state
I20260812 01:02:44.152797 18249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3: Bootstrap starting.
I20260812 01:02:44.174575 18249 log.cc:824] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3: Log is configured to *not* fsync() on all Append() calls
I20260812 01:02:44.186916 18249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 01:02:44.187513 18249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3: Bootstrap complete.
I20260812 01:02:44.197350 18249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1409835d19b84f5c84fef34e4cc8e8d3" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 32907 } }
I20260812 01:02:44.198290 18249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1409835d19b84f5c84fef34e4cc8e8d3, State: Initialized, Role: FOLLOWER
I20260812 01:02:44.198907 18249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1409835d19b84f5c84fef34e4cc8e8d3" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 32907 } }
I20260812 01:02:44.199193 18249 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:02:44.199337 18249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:02:44.199529 18249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [term 1 FOLLOWER]: Advancing to term 2
I20260812 01:02:44.203511 18249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1409835d19b84f5c84fef34e4cc8e8d3" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 32907 } }
I20260812 01:02:44.204101 18249 leader_election.cc:302] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1409835d19b84f5c84fef34e4cc8e8d3; no voters:
I20260812 01:02:44.204821 18249 leader_election.cc:288] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 01:02:44.204986 18258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 01:02:44.206151 18258 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [term 2 LEADER]: Becoming Leader. State: Replica: 1409835d19b84f5c84fef34e4cc8e8d3, State: Running, Role: LEADER
I20260812 01:02:44.206832 18258 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1409835d19b84f5c84fef34e4cc8e8d3" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 32907 } }
I20260812 01:02:44.207278 18249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:02:44.209689 18260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1409835d19b84f5c84fef34e4cc8e8d3. Latest consensus state: current_term: 2 leader_uuid: "1409835d19b84f5c84fef34e4cc8e8d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1409835d19b84f5c84fef34e4cc8e8d3" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 32907 } } }
I20260812 01:02:44.210086 18260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [sys.catalog]: This master's current role is: LEADER
I20260812 01:02:44.210749 18259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1409835d19b84f5c84fef34e4cc8e8d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1409835d19b84f5c84fef34e4cc8e8d3" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 32907 } } }
I20260812 01:02:44.211072 18259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [sys.catalog]: This master's current role is: LEADER
I20260812 01:02:44.224047 18264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:02:44.234283 18264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:02:44.236250 18264 catalog_manager.cc:1259] Loaded cluster ID: 746586bb7f194fd3ac33abe2533067a3
I20260812 01:02:44.236438 18264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:02:44.241870 18264 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:02:44.244016 18264 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3: Loaded TSK: 0
I20260812 01:02:44.268123 18264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3: Generated new TSK 1
I20260812 01:02:44.268529 18264 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 01:02:44.526068 18256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:02:44.526386 18256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:02:44.526562 18256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:02:44.534004 18256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:02:44.534168 18256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:02:44.534346 18256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 01:02:44.544325 18256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:39133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:41747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=33643
--tserver_master_addrs=127.21.201.126:32907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:02:44.545648 18256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:02:44.547017 18256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:02:44.562059 18296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:02:44.562142 18295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:02:44.562917 18302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:02:44.565512 18256 server_base.cc:1034] running on GCE node
I20260812 01:02:44.566221 18256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:02:44.567309 18256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:02:44.568571 18256 hybrid_clock.cc:648] HybridClock initialized: now 1786496564568519 us; error 48 us; skew 500 ppm
I20260812 01:02:44.568912 18256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:02:44 dist-test-slave-7f01 krb5kdc[17209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496564, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:02:44.572396 18256 init.cc:377] Logged in from keytab as kudu/127.21.201.65@KRBTEST.COM (short username kudu)
I20260812 01:02:44.574074 18256 webserver.cc:460] Webserver started at http://127.21.201.65:33643/ using document root <none> and password file <none>
I20260812 01:02:44.574816 18256 fs_manager.cc:359] Metadata directory not provided
I20260812 01:02:44.574971 18256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:02:44.580870 18256 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s
I20260812 01:02:44.583822 18320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:44.585247 18256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 01:02:44.585438 18256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "1c6362753ad54842abda04607ea7cbd5"
format_stamp: "Formatted at 2026-08-12 01:02:21 on dist-test-slave-7f01"
server_key: "8616e700ee2419079e083760906ada76"
server_key_iv: "ed27dcd2203d8f5567c847f828c40a1e"
server_key_version: "encryptionkey@0"
I20260812 01:02:44.586470 18256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:02:44.605533 18256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:02:44.610112 18256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:02:44.610415 18256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:02:44.611445 18256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:02:44.613032 18256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:02:44.613168 18256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:44.613312 18256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:02:44.613420 18256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:44.650669 18256 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:41747
I20260812 01:02:44.650766 18511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:41747 every 8 connection(s)
I20260812 01:02:44.652235 18256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 01:02:44.658249 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 18256
I20260812 01:02:44.658937 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:40855
--local_ip_for_outbound_sockets=127.21.201.66
--tserver_master_addrs=127.21.201.126:32907
--webserver_port=37485
--webserver_interface=127.21.201.66
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:39133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 01:02:44 dist-test-slave-7f01 krb5kdc[17209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496564, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:02:44.677469 18515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:44.655559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:56571 (local address 127.21.201.126:32907)
0812 01:02:44.656233 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:44.656246 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:44.657064 (+ 818us) server_negotiation.cc:407] Connection header received
0812 01:02:44.659366 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:44.659377 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:44.659690 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:44.660111 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:02:44.661742 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:44.663034 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:44.664912 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:44.665583 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:44.665806 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:44.669587 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:02:44.669596 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:02:44.669609 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:02:44.669648 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:02:44.672773 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:44.673587 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:44.673594 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:44.673600 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:44.673770 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:44.674237 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:44.674242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:44.674244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:44.674712 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:02:44.674948 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:44.675717 (+ 769us) server_negotiation.cc:299] Negotiation successful
0812 01:02:44.676156 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":324,"threads_started":1}
I20260812 01:02:44.677469 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:44.655878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.65:56571)
0812 01:02:44.656872 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:44.656921 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 01:02:44.658809 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:02:44.660267 (+ 1458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:02:44.660280 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:02:44.660959 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:02:44.661491 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:44.661509 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:44.663228 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:44.663232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:44.664759 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:44.664773 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:44.665857 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:44.665865 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:44.666165 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:44.666916 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:02:44.666945 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:02:44.669271 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:02:44.672947 (+ 3676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:02:44.672958 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:02:44.672975 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:02:44.673426 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:02:44.673894 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:02:44.673900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:02:44.673904 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:02:44.674100 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:02:44.674862 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:02:44.674871 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:02:44.675209 (+ 338us) client_negotiation.cc:769] Sending connection context
0812 01:02:44.675490 (+ 281us) client_negotiation.cc:240] Negotiation successful
0812 01:02:44.676075 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":429,"threads_started":1}
I20260812 01:02:44.680975 18512 heartbeater.cc:347] Connected to a master server at 127.21.201.126:32907
I20260812 01:02:44.681326 18512 heartbeater.cc:464] Registering TS with master...
I20260812 01:02:44.682097 18512 heartbeater.cc:511] Master 127.21.201.126:32907 requested a full tablet report, sending...
I20260812 01:02:44.684774 18186 ts_manager.cc:194] Registered new tserver with Master: 1c6362753ad54842abda04607ea7cbd5 (127.21.201.65:41747)
I20260812 01:02:44.687152 18186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.65@KRBTEST.COM'} at 127.21.201.65:56571
W20260812 01:02:45.012311 18517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:02:45.012614 18517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:02:45.012713 18517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:02:45.020162 18517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:02:45.020279 18517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:02:45.020454 18517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 01:02:45.030480 18517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:39133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:40855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=37485
--tserver_master_addrs=127.21.201.126:32907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:02:45.031849 18517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:02:45.033126 18517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:02:45.046880 18523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:02:45.047628 18526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:02:45.047781 18524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:02:45.053540 18517 server_base.cc:1034] running on GCE node
I20260812 01:02:45.054239 18517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:02:45.055292 18517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:02:45.056571 18517 hybrid_clock.cc:648] HybridClock initialized: now 1786496565056525 us; error 60 us; skew 500 ppm
I20260812 01:02:45.056936 18517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:02:45 dist-test-slave-7f01 krb5kdc[17209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496565, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:02:45.060662 18517 init.cc:377] Logged in from keytab as kudu/127.21.201.66@KRBTEST.COM (short username kudu)
I20260812 01:02:45.062330 18517 webserver.cc:460] Webserver started at http://127.21.201.66:37485/ using document root <none> and password file <none>
I20260812 01:02:45.063153 18517 fs_manager.cc:359] Metadata directory not provided
I20260812 01:02:45.063335 18517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:02:45.069796 18517 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 01:02:45.072818 18536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:45.074285 18517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 01:02:45.074505 18517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "cba9c8af143444528b5a7bc2c69e09a2"
format_stamp: "Formatted at 2026-08-12 01:02:21 on dist-test-slave-7f01"
server_key: "8dca22d8ec9ab7c87093148453b1ff41"
server_key_iv: "418a0bbe53e571e9fe241513faa6aacb"
server_key_version: "encryptionkey@0"
I20260812 01:02:45.075635 18517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:02:45.090472 18517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:02:45.095188 18517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:02:45.095511 18517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:02:45.096592 18517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:02:45.097992 18517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:02:45.098124 18517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:45.098268 18517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:02:45.098363 18517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:45.133337 18517 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:40855
I20260812 01:02:45.133373 18736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:40855 every 8 connection(s)
I20260812 01:02:45.135142 18517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 01:02:45.140195 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 18517
I20260812 01:02:45.141112 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:35975
--local_ip_for_outbound_sockets=127.21.201.67
--tserver_master_addrs=127.21.201.126:32907
--webserver_port=34809
--webserver_interface=127.21.201.67
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:39133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 01:02:45 dist-test-slave-7f01 krb5kdc[17209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496565, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:02:45.161358 18515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:45.137886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:47535 (local address 127.21.201.126:32907)
0812 01:02:45.138075 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:45.138080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:45.139428 (+ 1348us) server_negotiation.cc:407] Connection header received
0812 01:02:45.142455 (+ 3027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:45.142462 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:45.142567 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:45.142677 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:02:45.144911 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.146031 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.148626 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.149310 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.149439 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:45.154028 (+ 4589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:02:45.154034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:02:45.154038 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:02:45.154072 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:02:45.157676 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:45.158510 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:45.158518 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:45.158524 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:45.158687 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:45.159155 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:45.159163 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:45.159168 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:45.159610 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:02:45.159797 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:45.160579 (+ 782us) server_negotiation.cc:299] Negotiation successful
0812 01:02:45.160896 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 01:02:45.162041 18742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:45.138468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.66:47535)
0812 01:02:45.139265 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:45.139303 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 01:02:45.142075 (+ 2772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:02:45.142965 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:02:45.142978 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:02:45.143857 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:02:45.144704 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:45.144732 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.146212 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.146221 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:45.148450 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:45.148470 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.149567 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.149577 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:45.149904 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:45.150794 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:02:45.150829 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:02:45.153744 (+ 2915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:02:45.157859 (+ 4115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:02:45.157867 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:02:45.157879 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:02:45.158344 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:02:45.158802 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:02:45.158804 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:02:45.158807 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:02:45.158992 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:02:45.159799 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:02:45.159808 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:02:45.160161 (+ 353us) client_negotiation.cc:769] Sending connection context
0812 01:02:45.160388 (+ 227us) client_negotiation.cc:240] Negotiation successful
0812 01:02:45.160830 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":330,"threads_started":1}
I20260812 01:02:45.163636 18737 heartbeater.cc:347] Connected to a master server at 127.21.201.126:32907
I20260812 01:02:45.164004 18737 heartbeater.cc:464] Registering TS with master...
I20260812 01:02:45.164917 18737 heartbeater.cc:511] Master 127.21.201.126:32907 requested a full tablet report, sending...
I20260812 01:02:45.166652 18186 ts_manager.cc:194] Registered new tserver with Master: cba9c8af143444528b5a7bc2c69e09a2 (127.21.201.66:40855)
I20260812 01:02:45.167497 18186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.66@KRBTEST.COM'} at 127.21.201.66:47535
W20260812 01:02:45.497889 18743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:02:45.498198 18743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:02:45.498328 18743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:02:45.505692 18743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:02:45.505851 18743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:02:45.506054 18743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 01:02:45.516181 18743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:39133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:35975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=34809
--tserver_master_addrs=127.21.201.126:32907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:02:45.517540 18743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:02:45.518918 18743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:02:45.532437 18749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:02:45.532536 18751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:02:45.532426 18748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:02:45.533896 18743 server_base.cc:1034] running on GCE node
I20260812 01:02:45.534586 18743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:02:45.535701 18743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:02:45.536962 18743 hybrid_clock.cc:648] HybridClock initialized: now 1786496565536896 us; error 57 us; skew 500 ppm
I20260812 01:02:45.537298 18743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:02:45 dist-test-slave-7f01 krb5kdc[17209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496565, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:02:45.540783 18743 init.cc:377] Logged in from keytab as kudu/127.21.201.67@KRBTEST.COM (short username kudu)
I20260812 01:02:45.542471 18743 webserver.cc:460] Webserver started at http://127.21.201.67:34809/ using document root <none> and password file <none>
I20260812 01:02:45.543167 18743 fs_manager.cc:359] Metadata directory not provided
I20260812 01:02:45.543296 18743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:02:45.549023 18743 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 01:02:45.551841 18765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:45.553212 18743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:02:45.553366 18743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "762b216bab804e69ae171e5ae21e1bd1"
format_stamp: "Formatted at 2026-08-12 01:02:22 on dist-test-slave-7f01"
server_key: "508e56cdcb206f2d1a87853429c21f0b"
server_key_iv: "92f0e3a670001f51d1d3acf730a87bd0"
server_key_version: "encryptionkey@0"
I20260812 01:02:45.554349 18743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:02:45.568235 18743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:02:45.572813 18743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:02:45.573067 18743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:02:45.574007 18743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:02:45.575376 18743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:02:45.575466 18743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:45.575555 18743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:02:45.575615 18743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:02:45.611869 18743 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:35975
I20260812 01:02:45.611923 18960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:35975 every 8 connection(s)
I20260812 01:02:45.613400 18743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 01:02:45.617138 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 18743
Aug 12 01:02:45 dist-test-slave-7f01 krb5kdc[17209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496565, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:02:45.633679 18965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:45.618146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38306 (local address 127.21.201.126:32907)
0812 01:02:45.618658 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:45.618666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:45.619020 (+ 354us) server_negotiation.cc:407] Connection header received
0812 01:02:45.619253 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:45.619258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:45.619330 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:45.619451 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:02:45.621277 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.622139 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.623547 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.624064 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.624207 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:45.627362 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:02:45.627368 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:02:45.627371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:02:45.627403 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:02:45.630734 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:45.631462 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:45.631468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:45.631472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:45.631625 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:45.632084 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:45.632091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:45.632094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:45.632409 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:02:45.632731 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:45.633009 (+ 278us) server_negotiation.cc:299] Negotiation successful
0812 01:02:45.633249 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":264,"threads_started":1}
I20260812 01:02:45.635075 18515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:45.616123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:53007 (local address 127.21.201.126:32907)
0812 01:02:45.616356 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:45.616360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:45.617763 (+ 1403us) server_negotiation.cc:407] Connection header received
0812 01:02:45.620035 (+ 2272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:45.620041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:45.620124 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:45.620235 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:02:45.621963 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.622727 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.624509 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.624914 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.625067 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:45.628635 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:02:45.628641 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:02:45.628647 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:02:45.628691 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:02:45.631909 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:45.632665 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:45.632670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:45.632674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:45.632802 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:45.633275 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:45.633280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:45.633290 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:45.633651 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:02:45.633813 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:45.634508 (+ 695us) server_negotiation.cc:299] Negotiation successful
0812 01:02:45.634743 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 01:02:45.636168 18964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:45.616665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.67:53007)
0812 01:02:45.617569 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:45.617623 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 01:02:45.619711 (+ 2088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:02:45.620520 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:02:45.620528 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:02:45.621126 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:02:45.621754 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:45.621774 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.622898 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.622902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:45.624365 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:45.624378 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.625169 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.625178 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:45.625417 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:45.626050 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:02:45.626070 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:02:45.628404 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:02:45.632046 (+ 3642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:02:45.632056 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:02:45.632072 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:02:45.632530 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:02:45.632919 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:02:45.632924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:02:45.632928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:02:45.633144 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:02:45.633792 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:02:45.633800 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:02:45.634140 (+ 340us) client_negotiation.cc:769] Sending connection context
0812 01:02:45.634406 (+ 266us) client_negotiation.cc:240] Negotiation successful
0812 01:02:45.634963 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":340,"threads_started":1}
I20260812 01:02:45.637437 18962 heartbeater.cc:347] Connected to a master server at 127.21.201.126:32907
I20260812 01:02:45.637784 18962 heartbeater.cc:464] Registering TS with master...
I20260812 01:02:45.638554 18962 heartbeater.cc:511] Master 127.21.201.126:32907 requested a full tablet report, sending...
I20260812 01:02:45.640065 18186 ts_manager.cc:194] Registered new tserver with Master: 762b216bab804e69ae171e5ae21e1bd1 (127.21.201.67:35975)
I20260812 01:02:45.640939 18186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.67@KRBTEST.COM'} at 127.21.201.67:53007
I20260812 01:02:45.649202 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:02:45.656944 18515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:45.650330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38322 (local address 127.21.201.126:32907)
0812 01:02:45.650532 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:45.650539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:45.651069 (+ 530us) server_negotiation.cc:407] Connection header received
0812 01:02:45.651262 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:45.651270 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:45.651476 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:45.651649 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:02:45.652137 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.653288 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.654916 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.655546 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.655674 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:45.656195 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:02:45.656458 (+ 263us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0812 01:02:45.656753 (+ 295us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38322: token expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260812 01:02:45.657094 18515 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38322: token expired
W20260812 01:02:45.657843 18057 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.21.201.126:32907)
I20260812 01:02:45.670992 18515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:45.658514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38324 (local address 127.21.201.126:32907)
0812 01:02:45.658670 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:45.658674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:45.658690 (+ 16us) server_negotiation.cc:407] Connection header received
0812 01:02:45.658889 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:45.658901 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:45.658958 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:45.659054 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:02:45.660510 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.661216 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.662597 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.663121 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.663216 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:45.665956 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:02:45.665965 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:02:45.665971 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:02:45.666026 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:02:45.668606 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:45.669220 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:45.669224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:45.669226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:45.669350 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:45.669679 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:45.669682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:45.669684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:45.669982 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:02:45.670181 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:45.670447 (+ 266us) server_negotiation.cc:299] Negotiation successful
0812 01:02:45.670617 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 01:02:45.674463 18186 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:02:45.682282 18186 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:02:45.690865 18512 heartbeater.cc:503] Master 127.21.201.126:32907 was elected leader, sending a full tablet report...
I20260812 01:02:45.718755 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:45.696290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.21.201.65:41747)
0812 01:02:45.708083 (+ 11793us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:45.708096 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:45.708189 (+ 93us) server_negotiation.cc:407] Connection header received
0812 01:02:45.708486 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:45.708498 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:45.710643 (+ 2145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:45.710881 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:02:45.711661 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.713181 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.715619 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.717193 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.717348 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:45.717593 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:45.718208 (+ 615us) server_negotiation.cc:299] Negotiation successful
0812 01:02:45.718381 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11564,"thread_start_us":10643,"threads_started":1}
I20260812 01:02:45.718869 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:45.695544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:41747 (local address 127.0.0.1:42440)
0812 01:02:45.707549 (+ 12005us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:45.707575 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 01:02:45.707731 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:02:45.710910 (+ 3179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:02:45.710918 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:02:45.710950 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:02:45.711435 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:45.711449 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.713313 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.713321 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:45.715429 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:45.715452 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.717754 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.717761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:45.717978 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:45.718017 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 01:02:45.718172 (+ 155us) client_negotiation.cc:240] Negotiation successful
0812 01:02:45.718382 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11840,"mutex_wait_us":1,"thread_start_us":11660,"threads_started":1}
I20260812 01:02:45.719760 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:45.694750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:35975 (local address 127.0.0.1:58808)
0812 01:02:45.696269 (+ 1519us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:45.696305 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 01:02:45.696484 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:02:45.711940 (+ 15456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:02:45.711945 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:02:45.711970 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:02:45.712327 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:45.712341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.714347 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.714353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:45.716385 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:45.716398 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.718857 (+ 2459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.718864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:45.719008 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:45.719038 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 01:02:45.719117 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 01:02:45.719251 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1292,"mutex_wait_us":20,"thread_start_us":1155,"threads_started":1}
I20260812 01:02:45.720003 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:45.695072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58808 (local address 127.21.201.67:35975)
0812 01:02:45.711030 (+ 15958us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:45.711040 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:45.711075 (+ 35us) server_negotiation.cc:407] Connection header received
0812 01:02:45.711172 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:45.711180 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:45.711402 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:45.711609 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:02:45.712547 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.714178 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.716538 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.718274 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.718418 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:45.718633 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:45.719410 (+ 777us) server_negotiation.cc:299] Negotiation successful
0812 01:02:45.719568 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15760,"thread_start_us":16025,"threads_started":1}
I20260812 01:02:45.719960 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:45.696679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:40855 (local address 127.0.0.1:52442)
0812 01:02:45.709719 (+ 13040us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:45.709742 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 01:02:45.709899 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:02:45.712388 (+ 2489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:02:45.712394 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:02:45.712423 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:02:45.712759 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:45.712771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.714554 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.714568 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:45.716630 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:45.716648 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.718815 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.718825 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:45.719256 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:45.719286 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 01:02:45.719377 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 01:02:45.719500 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12861,"mutex_wait_us":38,"thread_start_us":13322,"threads_started":1}
I20260812 01:02:45.720896 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:45.697093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52442 (local address 127.21.201.66:40855)
0812 01:02:45.711784 (+ 14691us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:45.711793 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:45.711822 (+ 29us) server_negotiation.cc:407] Connection header received
0812 01:02:45.711926 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:45.711933 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:45.712150 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:45.712369 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:02:45.712941 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.714417 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.716819 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.718274 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.718409 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:45.718594 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:45.720086 (+ 1492us) server_negotiation.cc:299] Negotiation successful
0812 01:02:45.720270 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14474,"thread_start_us":14784,"threads_started":1}
I20260812 01:02:45.722960 18395 tablet_service.cc:1467] Processing CreateTablet for tablet 63a63c32f1cf42e4bc937c1a920db904 (DEFAULT_TABLE table=test-table [id=97f4e68eba824cbebcae462b5a0b631b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:02:45.724072 18619 tablet_service.cc:1467] Processing CreateTablet for tablet 63a63c32f1cf42e4bc937c1a920db904 (DEFAULT_TABLE table=test-table [id=97f4e68eba824cbebcae462b5a0b631b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:02:45.724072 18845 tablet_service.cc:1467] Processing CreateTablet for tablet 63a63c32f1cf42e4bc937c1a920db904 (DEFAULT_TABLE table=test-table [id=97f4e68eba824cbebcae462b5a0b631b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:02:45.724620 18395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a63c32f1cf42e4bc937c1a920db904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:02:45.725381 18845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a63c32f1cf42e4bc937c1a920db904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:02:45.725390 18619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a63c32f1cf42e4bc937c1a920db904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:02:45.737430 18992 tablet_bootstrap.cc:492] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5: Bootstrap starting.
I20260812 01:02:45.740875 18993 tablet_bootstrap.cc:492] T 63a63c32f1cf42e4bc937c1a920db904 P cba9c8af143444528b5a7bc2c69e09a2: Bootstrap starting.
I20260812 01:02:45.741067 18992 tablet_bootstrap.cc:654] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5: Neither blocks nor log segments found. Creating new log.
I20260812 01:02:45.741757 18998 tablet_bootstrap.cc:492] T 63a63c32f1cf42e4bc937c1a920db904 P 762b216bab804e69ae171e5ae21e1bd1: Bootstrap starting.
I20260812 01:02:45.743188 18992 log.cc:824] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5: Log is configured to *not* fsync() on all Append() calls
I20260812 01:02:45.745924 18998 tablet_bootstrap.cc:654] T 63a63c32f1cf42e4bc937c1a920db904 P 762b216bab804e69ae171e5ae21e1bd1: Neither blocks nor log segments found. Creating new log.
I20260812 01:02:45.746634 18993 tablet_bootstrap.cc:654] T 63a63c32f1cf42e4bc937c1a920db904 P cba9c8af143444528b5a7bc2c69e09a2: Neither blocks nor log segments found. Creating new log.
I20260812 01:02:45.747519 18998 log.cc:824] T 63a63c32f1cf42e4bc937c1a920db904 P 762b216bab804e69ae171e5ae21e1bd1: Log is configured to *not* fsync() on all Append() calls
I20260812 01:02:45.747639 18993 log.cc:824] T 63a63c32f1cf42e4bc937c1a920db904 P cba9c8af143444528b5a7bc2c69e09a2: Log is configured to *not* fsync() on all Append() calls
I20260812 01:02:45.750571 18993 tablet_bootstrap.cc:492] T 63a63c32f1cf42e4bc937c1a920db904 P cba9c8af143444528b5a7bc2c69e09a2: No bootstrap required, opened a new log
I20260812 01:02:45.750703 18992 tablet_bootstrap.cc:492] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5: No bootstrap required, opened a new log
I20260812 01:02:45.750692 18998 tablet_bootstrap.cc:492] T 63a63c32f1cf42e4bc937c1a920db904 P 762b216bab804e69ae171e5ae21e1bd1: No bootstrap required, opened a new log
I20260812 01:02:45.750906 18993 ts_tablet_manager.cc:1397] T 63a63c32f1cf42e4bc937c1a920db904 P cba9c8af143444528b5a7bc2c69e09a2: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260812 01:02:45.750977 18998 ts_tablet_manager.cc:1397] T 63a63c32f1cf42e4bc937c1a920db904 P 762b216bab804e69ae171e5ae21e1bd1: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260812 01:02:45.750977 18992 ts_tablet_manager.cc:1397] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20260812 01:02:45.759070 18993 raft_consensus.cc:348] T 63a63c32f1cf42e4bc937c1a920db904 P cba9c8af143444528b5a7bc2c69e09a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762b216bab804e69ae171e5ae21e1bd1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35975 } } peers { permanent_uuid: "cba9c8af143444528b5a7bc2c69e09a2" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40855 } } peers { permanent_uuid: "1c6362753ad54842abda04607ea7cbd5" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 41747 } }
I20260812 01:02:45.759560 18993 raft_consensus.cc:374] T 63a63c32f1cf42e4bc937c1a920db904 P cba9c8af143444528b5a7bc2c69e09a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:02:45.759778 18993 raft_consensus.cc:729] T 63a63c32f1cf42e4bc937c1a920db904 P cba9c8af143444528b5a7bc2c69e09a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cba9c8af143444528b5a7bc2c69e09a2, State: Initialized, Role: FOLLOWER
I20260812 01:02:45.760483 18993 consensus_queue.cc:260] T 63a63c32f1cf42e4bc937c1a920db904 P cba9c8af143444528b5a7bc2c69e09a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762b216bab804e69ae171e5ae21e1bd1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35975 } } peers { permanent_uuid: "cba9c8af143444528b5a7bc2c69e09a2" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40855 } } peers { permanent_uuid: "1c6362753ad54842abda04607ea7cbd5" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 41747 } }
I20260812 01:02:45.761675 18737 heartbeater.cc:503] Master 127.21.201.126:32907 was elected leader, sending a full tablet report...
I20260812 01:02:45.762338 18992 raft_consensus.cc:348] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762b216bab804e69ae171e5ae21e1bd1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35975 } } peers { permanent_uuid: "cba9c8af143444528b5a7bc2c69e09a2" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40855 } } peers { permanent_uuid: "1c6362753ad54842abda04607ea7cbd5" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 41747 } }
I20260812 01:02:45.762866 18992 raft_consensus.cc:374] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:02:45.762989 18992 raft_consensus.cc:729] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c6362753ad54842abda04607ea7cbd5, State: Initialized, Role: FOLLOWER
I20260812 01:02:45.762730 18998 raft_consensus.cc:348] T 63a63c32f1cf42e4bc937c1a920db904 P 762b216bab804e69ae171e5ae21e1bd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762b216bab804e69ae171e5ae21e1bd1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35975 } } peers { permanent_uuid: "cba9c8af143444528b5a7bc2c69e09a2" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40855 } } peers { permanent_uuid: "1c6362753ad54842abda04607ea7cbd5" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 41747 } }
I20260812 01:02:45.763213 18998 raft_consensus.cc:374] T 63a63c32f1cf42e4bc937c1a920db904 P 762b216bab804e69ae171e5ae21e1bd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:02:45.763337 18998 raft_consensus.cc:729] T 63a63c32f1cf42e4bc937c1a920db904 P 762b216bab804e69ae171e5ae21e1bd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 762b216bab804e69ae171e5ae21e1bd1, State: Initialized, Role: FOLLOWER
I20260812 01:02:45.763728 18993 ts_tablet_manager.cc:1428] T 63a63c32f1cf42e4bc937c1a920db904 P cba9c8af143444528b5a7bc2c69e09a2: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260812 01:02:45.763654 18992 consensus_queue.cc:260] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762b216bab804e69ae171e5ae21e1bd1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35975 } } peers { permanent_uuid: "cba9c8af143444528b5a7bc2c69e09a2" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40855 } } peers { permanent_uuid: "1c6362753ad54842abda04607ea7cbd5" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 41747 } }
I20260812 01:02:45.763932 18998 consensus_queue.cc:260] T 63a63c32f1cf42e4bc937c1a920db904 P 762b216bab804e69ae171e5ae21e1bd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762b216bab804e69ae171e5ae21e1bd1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35975 } } peers { permanent_uuid: "cba9c8af143444528b5a7bc2c69e09a2" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40855 } } peers { permanent_uuid: "1c6362753ad54842abda04607ea7cbd5" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 41747 } }
I20260812 01:02:45.765825 18992 ts_tablet_manager.cc:1428] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260812 01:02:45.767249 18962 heartbeater.cc:503] Master 127.21.201.126:32907 was elected leader, sending a full tablet report...
I20260812 01:02:45.767930 18998 ts_tablet_manager.cc:1428] T 63a63c32f1cf42e4bc937c1a920db904 P 762b216bab804e69ae171e5ae21e1bd1: Time spent starting tablet: real 0.017s user 0.018s sys 0.000s
W20260812 01:02:45.865185 18963 tablet.cc:2369] T 63a63c32f1cf42e4bc937c1a920db904 P 762b216bab804e69ae171e5ae21e1bd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:02:45.886834 19004 raft_consensus.cc:482] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 01:02:45.887355 18741 tablet.cc:2369] T 63a63c32f1cf42e4bc937c1a920db904 P cba9c8af143444528b5a7bc2c69e09a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:02:45.887198 19004 raft_consensus.cc:504] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762b216bab804e69ae171e5ae21e1bd1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35975 } } peers { permanent_uuid: "cba9c8af143444528b5a7bc2c69e09a2" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40855 } } peers { permanent_uuid: "1c6362753ad54842abda04607ea7cbd5" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 41747 } }
I20260812 01:02:45.889117 19004 leader_election.cc:288] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 762b216bab804e69ae171e5ae21e1bd1 (127.21.201.67:35975), cba9c8af143444528b5a7bc2c69e09a2 (127.21.201.66:40855)
I20260812 01:02:45.897656 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:45.889553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:39785 (local address 127.21.201.67:35975)
0812 01:02:45.889811 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:45.889822 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:45.890446 (+ 624us) server_negotiation.cc:407] Connection header received
0812 01:02:45.890682 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:45.890706 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:45.890798 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:45.890893 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:02:45.891866 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.892686 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.895521 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.896544 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.896777 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:45.896868 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:45.897191 (+ 323us) server_negotiation.cc:299] Negotiation successful
0812 01:02:45.897340 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 01:02:45.897743 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:45.889563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:35975 (local address 127.21.201.65:39785)
0812 01:02:45.890331 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:45.890348 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:02:45.890526 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:02:45.891140 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:02:45.891145 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:02:45.891171 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:02:45.891648 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:45.891661 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.892976 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.892981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:45.895337 (+ 2356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:45.895352 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.896787 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.896794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:45.897006 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:45.897034 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 01:02:45.897157 (+ 123us) client_negotiation.cc:240] Negotiation successful
0812 01:02:45.897304 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":429,"threads_started":1}
I20260812 01:02:45.898403 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:45.890681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:45819 (local address 127.21.201.66:40855)
0812 01:02:45.890928 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:45.890934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:45.891347 (+ 413us) server_negotiation.cc:407] Connection header received
0812 01:02:45.891563 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:45.891571 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:45.891668 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:45.891786 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:02:45.893497 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.894731 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.896509 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.897322 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.897510 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:45.897596 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:45.897976 (+ 380us) server_negotiation.cc:299] Negotiation successful
0812 01:02:45.898112 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 01:02:45.899025 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:45.890521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:40855 (local address 127.21.201.65:45819)
0812 01:02:45.891244 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:45.891258 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 01:02:45.891419 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:02:45.892868 (+ 1449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:02:45.892872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:02:45.892896 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:02:45.893277 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:45.893286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.894873 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.894876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:45.896342 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:02:45.896354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.897584 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.897590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:02:45.897738 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:45.897797 (+ 59us) client_negotiation.cc:769] Sending connection context
0812 01:02:45.898590 (+ 793us) client_negotiation.cc:240] Negotiation successful
0812 01:02:45.898736 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"spinlock_wait_cycles":16768,"thread_start_us":412,"threads_started":1}
I20260812 01:02:45.899127 18890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a63c32f1cf42e4bc937c1a920db904" candidate_uuid: "1c6362753ad54842abda04607ea7cbd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762b216bab804e69ae171e5ae21e1bd1" is_pre_election: true
I20260812 01:02:45.899729 18890 raft_consensus.cc:2454] T 63a63c32f1cf42e4bc937c1a920db904 P 762b216bab804e69ae171e5ae21e1bd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c6362753ad54842abda04607ea7cbd5 in term 0.
I20260812 01:02:45.899972 18654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a63c32f1cf42e4bc937c1a920db904" candidate_uuid: "1c6362753ad54842abda04607ea7cbd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cba9c8af143444528b5a7bc2c69e09a2" is_pre_election: true
I20260812 01:02:45.900473 18654 raft_consensus.cc:2454] T 63a63c32f1cf42e4bc937c1a920db904 P cba9c8af143444528b5a7bc2c69e09a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c6362753ad54842abda04607ea7cbd5 in term 0.
I20260812 01:02:45.900607 18325 leader_election.cc:302] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c6362753ad54842abda04607ea7cbd5, 762b216bab804e69ae171e5ae21e1bd1; no voters:
I20260812 01:02:45.901019 19004 raft_consensus.cc:2781] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:02:45.901196 19004 raft_consensus.cc:482] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:02:45.901332 19004 raft_consensus.cc:3037] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:02:45.903983 19004 raft_consensus.cc:504] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762b216bab804e69ae171e5ae21e1bd1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35975 } } peers { permanent_uuid: "cba9c8af143444528b5a7bc2c69e09a2" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40855 } } peers { permanent_uuid: "1c6362753ad54842abda04607ea7cbd5" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 41747 } }
I20260812 01:02:45.904727 19004 leader_election.cc:288] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5 [CANDIDATE]: Term 1 election: Requested vote from peers 762b216bab804e69ae171e5ae21e1bd1 (127.21.201.67:35975), cba9c8af143444528b5a7bc2c69e09a2 (127.21.201.66:40855)
W20260812 01:02:45.904937 18514 tablet.cc:2369] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:02:45.905167 18890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a63c32f1cf42e4bc937c1a920db904" candidate_uuid: "1c6362753ad54842abda04607ea7cbd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762b216bab804e69ae171e5ae21e1bd1"
I20260812 01:02:45.905256 18654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a63c32f1cf42e4bc937c1a920db904" candidate_uuid: "1c6362753ad54842abda04607ea7cbd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cba9c8af143444528b5a7bc2c69e09a2"
I20260812 01:02:45.905416 18890 raft_consensus.cc:3037] T 63a63c32f1cf42e4bc937c1a920db904 P 762b216bab804e69ae171e5ae21e1bd1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:02:45.905510 18654 raft_consensus.cc:3037] T 63a63c32f1cf42e4bc937c1a920db904 P cba9c8af143444528b5a7bc2c69e09a2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:02:45.908277 18890 raft_consensus.cc:2454] T 63a63c32f1cf42e4bc937c1a920db904 P 762b216bab804e69ae171e5ae21e1bd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c6362753ad54842abda04607ea7cbd5 in term 1.
I20260812 01:02:45.908316 18654 raft_consensus.cc:2454] T 63a63c32f1cf42e4bc937c1a920db904 P cba9c8af143444528b5a7bc2c69e09a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c6362753ad54842abda04607ea7cbd5 in term 1.
I20260812 01:02:45.908751 18325 leader_election.cc:302] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c6362753ad54842abda04607ea7cbd5, 762b216bab804e69ae171e5ae21e1bd1; no voters:
I20260812 01:02:45.909067 19004 raft_consensus.cc:2781] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:02:45.909554 19004 raft_consensus.cc:686] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5 [term 1 LEADER]: Becoming Leader. State: Replica: 1c6362753ad54842abda04607ea7cbd5, State: Running, Role: LEADER
I20260812 01:02:45.910095 19004 consensus_queue.cc:237] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762b216bab804e69ae171e5ae21e1bd1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35975 } } peers { permanent_uuid: "cba9c8af143444528b5a7bc2c69e09a2" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40855 } } peers { permanent_uuid: "1c6362753ad54842abda04607ea7cbd5" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 41747 } }
I20260812 01:02:45.915064 18186 catalog_manager.cc:5495] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c6362753ad54842abda04607ea7cbd5 (127.21.201.65). New cstate: current_term: 1 leader_uuid: "1c6362753ad54842abda04607ea7cbd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762b216bab804e69ae171e5ae21e1bd1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cba9c8af143444528b5a7bc2c69e09a2" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c6362753ad54842abda04607ea7cbd5" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 41747 } health_report { overall_health: HEALTHY } } }
I20260812 01:02:45.944387 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:45.938454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42452 (local address 127.21.201.65:41747)
0812 01:02:45.938680 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:45.938686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:45.938708 (+ 22us) server_negotiation.cc:407] Connection header received
0812 01:02:45.938926 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:45.938931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:45.939013 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:45.939174 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:02:45.939587 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.940418 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.942039 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.942453 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:45.942560 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:45.943295 (+ 735us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:02:45.943877 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:02:45.943993 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:45.944108 (+ 115us) server_negotiation.cc:299] Negotiation successful
0812 01:02:45.944179 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 01:02:45.953089 18890 raft_consensus.cc:1261] T 63a63c32f1cf42e4bc937c1a920db904 P 762b216bab804e69ae171e5ae21e1bd1 [term 1 FOLLOWER]: Refusing update from remote peer 1c6362753ad54842abda04607ea7cbd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:02:45.953089 18654 raft_consensus.cc:1261] T 63a63c32f1cf42e4bc937c1a920db904 P cba9c8af143444528b5a7bc2c69e09a2 [term 1 FOLLOWER]: Refusing update from remote peer 1c6362753ad54842abda04607ea7cbd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:02:45.954190 19013 consensus_queue.cc:1035] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cba9c8af143444528b5a7bc2c69e09a2" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 40855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:02:45.954569 19004 consensus_queue.cc:1035] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "762b216bab804e69ae171e5ae21e1bd1" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 35975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:02:45.969080 19027 mvcc.cc:205] Tried to move back new op lower bound from 7317489934132252672 to 7317489933972140032. Current Snapshot: MvccSnapshot[applied={T|T < 7317489934132252672}]
I20260812 01:02:45.982544 18185 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:02:45.982862 18185 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:02:45.986789 18185 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3: Sending DeleteTablet for 3 replicas of tablet 63a63c32f1cf42e4bc937c1a920db904
I20260812 01:02:45.987988 18845 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a63c32f1cf42e4bc937c1a920db904 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:02:45 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:58808
I20260812 01:02:45.988107 18619 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a63c32f1cf42e4bc937c1a920db904 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:02:45 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:52442
I20260812 01:02:45.988130 18395 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a63c32f1cf42e4bc937c1a920db904 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:02:45 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:42440
I20260812 01:02:45.990653 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 18256
I20260812 01:02:46.002084 19045 tablet_replica.cc:330] T 63a63c32f1cf42e4bc937c1a920db904 P 762b216bab804e69ae171e5ae21e1bd1: stopping tablet replica
I20260812 01:02:46.003226 19045 raft_consensus.cc:2229] T 63a63c32f1cf42e4bc937c1a920db904 P 762b216bab804e69ae171e5ae21e1bd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:02:46.004427 19045 raft_consensus.cc:2258] T 63a63c32f1cf42e4bc937c1a920db904 P 762b216bab804e69ae171e5ae21e1bd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:02:46.007241 19046 tablet_replica.cc:330] T 63a63c32f1cf42e4bc937c1a920db904 P cba9c8af143444528b5a7bc2c69e09a2: stopping tablet replica
I20260812 01:02:46.007542 19049 tablet_replica.cc:330] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5: stopping tablet replica
I20260812 01:02:46.007906 19046 raft_consensus.cc:2229] T 63a63c32f1cf42e4bc937c1a920db904 P cba9c8af143444528b5a7bc2c69e09a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:02:46.008096 19049 raft_consensus.cc:2229] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5 [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:02:46.008512 19046 raft_consensus.cc:2258] T 63a63c32f1cf42e4bc937c1a920db904 P cba9c8af143444528b5a7bc2c69e09a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:02:46.008944 19049 raft_consensus.cc:2258] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:02:46.011353 19045 ts_tablet_manager.cc:1905] T 63a63c32f1cf42e4bc937c1a920db904 P 762b216bab804e69ae171e5ae21e1bd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:02:46.012811 19049 ts_tablet_manager.cc:1905] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:02:46.017130 19046 ts_tablet_manager.cc:1905] T 63a63c32f1cf42e4bc937c1a920db904 P cba9c8af143444528b5a7bc2c69e09a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:02:46.019959 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:45.992629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42466 (local address 127.21.201.65:41747)
0812 01:02:45.992876 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:45.992885 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:45.992909 (+ 24us) server_negotiation.cc:407] Connection header received
0812 01:02:45.992982 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:45.992990 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:45.993102 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:45.993226 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:02:45.995862 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:45.997581 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:46.000431 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:46.001444 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:46.001604 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:46.006950 (+ 5346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:02:46.006967 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:02:46.006986 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:02:46.007045 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:02:46.014449 (+ 7404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:46.015239 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:46.015252 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:46.015256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:46.015445 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:46.017346 (+ 1901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:46.017353 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:46.017358 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:46.017836 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:02:46.018226 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:46.019225 (+ 999us) server_negotiation.cc:299] Negotiation successful
0812 01:02:46.019492 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 01:02:46.020952 18500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:02:46.020992 19045 ts_tablet_manager.cc:1918] T 63a63c32f1cf42e4bc937c1a920db904 P 762b216bab804e69ae171e5ae21e1bd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:02:46.021162 19045 log.cc:1196] T 63a63c32f1cf42e4bc937c1a920db904 P 762b216bab804e69ae171e5ae21e1bd1: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/63a63c32f1cf42e4bc937c1a920db904
I20260812 01:02:46.021751 19045 ts_tablet_manager.cc:1939] T 63a63c32f1cf42e4bc937c1a920db904 P 762b216bab804e69ae171e5ae21e1bd1: Deleting consensus metadata
I20260812 01:02:46.023854 18163 catalog_manager.cc:4841] TS 762b216bab804e69ae171e5ae21e1bd1 (127.21.201.67:35975): tablet 63a63c32f1cf42e4bc937c1a920db904 (table test-table [id=97f4e68eba824cbebcae462b5a0b631b]) successfully deleted
I20260812 01:02:46.027563 19046 ts_tablet_manager.cc:1918] T 63a63c32f1cf42e4bc937c1a920db904 P cba9c8af143444528b5a7bc2c69e09a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:02:46.027726 19046 log.cc:1196] T 63a63c32f1cf42e4bc937c1a920db904 P cba9c8af143444528b5a7bc2c69e09a2: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/63a63c32f1cf42e4bc937c1a920db904
I20260812 01:02:46.028401 19046 ts_tablet_manager.cc:1939] T 63a63c32f1cf42e4bc937c1a920db904 P cba9c8af143444528b5a7bc2c69e09a2: Deleting consensus metadata
I20260812 01:02:46.029910 18163 catalog_manager.cc:4841] TS cba9c8af143444528b5a7bc2c69e09a2 (127.21.201.66:40855): tablet 63a63c32f1cf42e4bc937c1a920db904 (table test-table [id=97f4e68eba824cbebcae462b5a0b631b]) successfully deleted
I20260812 01:02:46.127050 19049 ts_tablet_manager.cc:1918] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:02:46.127703 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 18256
I20260812 01:02:46.127890 19049 log.cc:1196] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/wal/wals/63a63c32f1cf42e4bc937c1a920db904
W20260812 01:02:46.157442 18166 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.201.65:41747 (error 108)
I20260812 01:02:46.157699 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 18517
I20260812 01:02:46.159363 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:46.158901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:41747 (local address 127.0.0.1:42476)
0812 01:02:46.159115 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:46.159218 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:41747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260812 01:02:46.159585 18166 catalog_manager.cc:4584] TS 1c6362753ad54842abda04607ea7cbd5 (127.21.201.65:41747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 63a63c32f1cf42e4bc937c1a920db904: Network error: Client connection negotiation failed: client connection to 127.21.201.65:41747: connect: Connection refused (error 111)
I20260812 01:02:46.172688 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:46.158630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52446 (local address 127.21.201.66:40855)
0812 01:02:46.158815 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:46.158821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:46.158843 (+ 22us) server_negotiation.cc:407] Connection header received
0812 01:02:46.158951 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:46.158957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:46.159100 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:46.159236 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:02:46.160796 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:46.161576 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:46.162973 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:46.163390 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:46.163530 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:46.166618 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:02:46.166630 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:02:46.166645 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:02:46.166680 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:02:46.169725 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:46.170434 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:46.170451 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:46.170454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:46.170583 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:46.170987 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:46.170993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:46.170998 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:46.171408 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:02:46.171736 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:46.172101 (+ 365us) server_negotiation.cc:299] Negotiation successful
0812 01:02:46.172333 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 01:02:46.173410 18729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:02:46.213675 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:46.213059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:41747 (local address 127.0.0.1:42490)
0812 01:02:46.213356 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:46.213477 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:41747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260812 01:02:46.214838 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:46.214450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:41747 (local address 127.0.0.1:42506)
0812 01:02:46.214629 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:46.214723 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:41747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260812 01:02:46.268091 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 18517
I20260812 01:02:46.276158 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:46.275518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:41747 (local address 127.0.0.1:42516)
0812 01:02:46.275807 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:46.275931 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:41747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260812 01:02:46.277369 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:46.276909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:41747 (local address 127.0.0.1:42524)
0812 01:02:46.277101 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:46.277201 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:41747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260812 01:02:46.292279 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 18743
I20260812 01:02:46.306851 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:46.293068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58824 (local address 127.21.201.67:35975)
0812 01:02:46.293278 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 01:02:46.293284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:02:46.293306 (+ 22us) server_negotiation.cc:407] Connection header received
0812 01:02:46.293522 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:02:46.293527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:02:46.293597 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:02:46.293711 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:02:46.295147 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:46.295981 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:46.297323 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:02:46.297740 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:02:46.297875 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:02:46.300791 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:02:46.300807 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:02:46.300825 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:02:46.300877 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:02:46.303861 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:46.304609 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:46.304616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:46.304618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:46.304775 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:02:46.305190 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:02:46.305195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:02:46.305200 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:02:46.305561 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:02:46.305892 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0812 01:02:46.306222 (+ 330us) server_negotiation.cc:299] Negotiation successful
0812 01:02:46.306463 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 01:02:46.307523 18954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:02:46.358217 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:46.357581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:41747 (local address 127.0.0.1:42532)
0812 01:02:46.357852 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:46.357975 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:41747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260812 01:02:46.359383 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:46.358944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:41747 (local address 127.0.0.1:42538)
0812 01:02:46.359166 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:46.359253 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:41747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260812 01:02:46.401384 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 18743
I20260812 01:02:46.430866 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 18133
I20260812 01:02:46.432233 18243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:02:46.554374 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:02:46.553690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:41747 (local address 127.0.0.1:42548)
0812 01:02:46.554031 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0812 01:02:46.554160 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:41747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
I20260812 01:02:46.554428 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 18133
I20260812 01:03:06.580155 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:32907
--webserver_interface=127.21.201.126
--webserver_port=38777
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:39133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:32907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:03:06.942185 19055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:06.942546 19055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:06.942713 19055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:06.950196 19055 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 01:03:06.950343 19055 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 01:03:06.950425 19055 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 01:03:06.950528 19055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:03:06.950637 19055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:06.950766 19055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:06.950870 19055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:03:06.950954 19055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:03:06.960999 19055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:39133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:32907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:32907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=38777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:06.962393 19055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:06.963825 19055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:06.976732 19065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:06.976807 19067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:06.977188 19064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:06.977499 19055 server_base.cc:1034] running on GCE node
I20260812 01:03:06.978492 19055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:06.980001 19055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:06.981248 19055 hybrid_clock.cc:648] HybridClock initialized: now 1786496586981175 us; error 52 us; skew 500 ppm
I20260812 01:03:06.981590 19055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:03:06 dist-test-slave-7f01 krb5kdc[17209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496586, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:03:06.989712 19055 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 01:03:06.991271 19055 webserver.cc:460] Webserver started at http://127.21.201.126:38777/ using document root <none> and password file <none>
I20260812 01:03:06.992049 19055 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:06.992146 19055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:06.997905 19055 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260812 01:03:07.000788 19078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:07.002178 19055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 01:03:07.002329 19055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "1409835d19b84f5c84fef34e4cc8e8d3"
format_stamp: "Formatted at 2026-08-12 01:02:20 on dist-test-slave-7f01"
server_key: "e2b99432c1792435d918d9268787e6de"
server_key_iv: "25f718051a7f4176395a81a6be28962f"
server_key_version: "encryptionkey@0"
I20260812 01:03:07.003345 19055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:07.035341 19055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:07.039690 19055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:07.039985 19055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:07.060242 19055 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:32907
I20260812 01:03:07.060246 19166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:32907 every 8 connection(s)
I20260812 01:03:07.061764 19055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:03:07.067564 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 19055
I20260812 01:03:07.068706 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:41747
--local_ip_for_outbound_sockets=127.21.201.65
--tserver_master_addrs=127.21.201.126:32907
--webserver_port=33643
--webserver_interface=127.21.201.65
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:39133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:03:07.069674 19167 sys_catalog.cc:263] Verifying existing consensus state
I20260812 01:03:07.074944 19167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3: Bootstrap starting.
I20260812 01:03:07.100432 19167 log.cc:824] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:07.118340 19167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 01:03:07.118992 19167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3: Bootstrap complete.
I20260812 01:03:07.128917 19167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1409835d19b84f5c84fef34e4cc8e8d3" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 32907 } }
I20260812 01:03:07.129866 19167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1409835d19b84f5c84fef34e4cc8e8d3, State: Initialized, Role: FOLLOWER
I20260812 01:03:07.130517 19167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1409835d19b84f5c84fef34e4cc8e8d3" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 32907 } }
I20260812 01:03:07.130786 19167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:03:07.130915 19167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:03:07.131129 19167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [term 2 FOLLOWER]: Advancing to term 3
I20260812 01:03:07.135208 19167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1409835d19b84f5c84fef34e4cc8e8d3" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 32907 } }
I20260812 01:03:07.135723 19167 leader_election.cc:302] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1409835d19b84f5c84fef34e4cc8e8d3; no voters:
I20260812 01:03:07.136528 19167 leader_election.cc:288] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 01:03:07.136690 19175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 01:03:07.137848 19175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [term 3 LEADER]: Becoming Leader. State: Replica: 1409835d19b84f5c84fef34e4cc8e8d3, State: Running, Role: LEADER
I20260812 01:03:07.138523 19175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1409835d19b84f5c84fef34e4cc8e8d3" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 32907 } }
I20260812 01:03:07.138859 19167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:03:07.141185 19177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1409835d19b84f5c84fef34e4cc8e8d3. Latest consensus state: current_term: 3 leader_uuid: "1409835d19b84f5c84fef34e4cc8e8d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1409835d19b84f5c84fef34e4cc8e8d3" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 32907 } } }
I20260812 01:03:07.141654 19177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [sys.catalog]: This master's current role is: LEADER
I20260812 01:03:07.142292 19176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1409835d19b84f5c84fef34e4cc8e8d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1409835d19b84f5c84fef34e4cc8e8d3" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 32907 } } }
I20260812 01:03:07.142596 19176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3 [sys.catalog]: This master's current role is: LEADER
I20260812 01:03:07.142620 19183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:03:07.154601 19183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:03:07.156283 19183 catalog_manager.cc:1259] Loaded cluster ID: 746586bb7f194fd3ac33abe2533067a3
I20260812 01:03:07.156473 19183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:03:07.160601 19183 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:03:07.162681 19183 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3: Loaded TSK: 0
I20260812 01:03:07.162926 19183 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1409835d19b84f5c84fef34e4cc8e8d3: Loaded TSK: 1
F20260812 01:03:07.164173 19183 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd8fc4df9ad google::LogMessage::Fail() at ??:0
@ 0x7fd8fc4e288a google::LogMessage::SendToLog() at ??:0
@ 0x7fd8fc4df4de google::LogMessage::Flush() at ??:0
@ 0x7fd8fc4e0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd9002437d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd90023273e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fd911f0e1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fd911f0d783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fd911f992a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fd911f990c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fd911fe35e9 std::function<>::operator()() at ??:0
@ 0x7fd911f11cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fd911f0fc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd911f97610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fd911f97439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd8ff12b345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd8ff12e91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fd8ff12e799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd8ff109eaf kudu::Thread::SuperviseThread() at ??:0
I20260812 01:03:07.281726 19215 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fd9023e36db start_thread at ??:0
@ 0x7fd8fb37961f clone at ??:0
W20260812 01:03:07.556496 19171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:07.556794 19171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:07.556897 19171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:07.564987 19171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:07.565104 19171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:07.565234 19171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 01:03:07.574946 19171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:39133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:41747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=33643
--tserver_master_addrs=127.21.201.126:32907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:07.576221 19171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:07.577533 19171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:07.591317 19248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:07.591593 19250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:07.591948 19252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:07.592963 19171 server_base.cc:1034] running on GCE node
I20260812 01:03:07.593665 19171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:07.594633 19171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:07.595860 19171 hybrid_clock.cc:648] HybridClock initialized: now 1786496587595799 us; error 40 us; skew 500 ppm
I20260812 01:03:07.596189 19171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:03:07 dist-test-slave-7f01 krb5kdc[17209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496587, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:03:07.599608 19171 init.cc:377] Logged in from keytab as kudu/127.21.201.65@KRBTEST.COM (short username kudu)
I20260812 01:03:07.601260 19171 webserver.cc:460] Webserver started at http://127.21.201.65:33643/ using document root <none> and password file <none>
I20260812 01:03:07.601939 19171 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:07.602047 19171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:07.607518 19171 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 01:03:07.610453 19266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:07.612088 19171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:03:07.612325 19171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "1c6362753ad54842abda04607ea7cbd5"
format_stamp: "Formatted at 2026-08-12 01:02:21 on dist-test-slave-7f01"
server_key: "8616e700ee2419079e083760906ada76"
server_key_iv: "ed27dcd2203d8f5567c847f828c40a1e"
server_key_version: "encryptionkey@0"
I20260812 01:03:07.613452 19171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:07.632530 19171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:07.637037 19171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:07.637365 19171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:07.638415 19171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:03:07.640383 19280 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260812 01:03:07.646059 19280 ts_tablet_manager.cc:1852] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260812 01:03:07.647145 19280 ts_tablet_manager.cc:1905] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:03:07.654778 19280 ts_tablet_manager.cc:1918] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:03:07.655030 19280 log.cc:1196] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/wal/wals/63a63c32f1cf42e4bc937c1a920db904
I20260812 01:03:07.655562 19280 ts_tablet_manager.cc:1939] T 63a63c32f1cf42e4bc937c1a920db904 P 1c6362753ad54842abda04607ea7cbd5: Deleting consensus metadata
I20260812 01:03:07.656033 19171 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260812 01:03:07.656190 19171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.000s sys 0.000s
I20260812 01:03:07.656353 19171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:03:07.656445 19171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:07.695050 19171 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:41747
I20260812 01:03:07.695159 19471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:41747 every 8 connection(s)
I20260812 01:03:07.696518 19171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 01:03:07.699574 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 19171
I20260812 01:03:07.700623 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:40855
--local_ip_for_outbound_sockets=127.21.201.66
--tserver_master_addrs=127.21.201.126:32907
--webserver_port=37485
--webserver_interface=127.21.201.66
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:39133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:03:07.702535 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:07.700203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.65:58145)
0812 01:03:07.701346 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:07.701637 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":421,"threads_started":1}
W20260812 01:03:07.703337 19472 heartbeater.cc:644] Failed to heartbeat to 127.21.201.126:32907 (0 consecutive failures): Network error: Failed to ping master at 127.21.201.126:32907: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
I20260812 01:03:07.704478 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:07.704077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.65:54709)
0812 01:03:07.704258 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:07.704349 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260812 01:03:07.705538 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:07.705178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.65:48297)
0812 01:03:07.705322 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:07.705401 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 01:03:07.705842 19472 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 01:03:08.064538 19476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:08.064834 19476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:08.064935 19476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:08.077221 19476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:08.077503 19476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:08.077833 19476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 01:03:08.094344 19476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:39133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:40855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=37485
--tserver_master_addrs=127.21.201.126:32907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:08.096343 19476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:08.098354 19476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:08.117074 19485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:08.117179 19486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:08.117233 19490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:08.118592 19476 server_base.cc:1034] running on GCE node
I20260812 01:03:08.119475 19476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:08.120891 19476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:08.122130 19476 hybrid_clock.cc:648] HybridClock initialized: now 1786496588122030 us; error 73 us; skew 500 ppm
I20260812 01:03:08.122536 19476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:03:08 dist-test-slave-7f01 krb5kdc[17209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496588, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:03:08.125980 19476 init.cc:377] Logged in from keytab as kudu/127.21.201.66@KRBTEST.COM (short username kudu)
I20260812 01:03:08.127624 19476 webserver.cc:460] Webserver started at http://127.21.201.66:37485/ using document root <none> and password file <none>
I20260812 01:03:08.128756 19476 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:08.128916 19476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:08.135679 19476 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260812 01:03:08.138576 19499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:08.140005 19476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:03:08.140197 19476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "cba9c8af143444528b5a7bc2c69e09a2"
format_stamp: "Formatted at 2026-08-12 01:02:21 on dist-test-slave-7f01"
server_key: "8dca22d8ec9ab7c87093148453b1ff41"
server_key_iv: "418a0bbe53e571e9fe241513faa6aacb"
server_key_version: "encryptionkey@0"
I20260812 01:03:08.141288 19476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:08.174772 19476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:08.179715 19476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:08.180133 19476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:08.181296 19476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:03:08.182816 19476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:03:08.182919 19476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:08.183058 19476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:03:08.183171 19476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:08.220781 19476 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:40855
I20260812 01:03:08.220882 19695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:40855 every 8 connection(s)
I20260812 01:03:08.222437 19476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 01:03:08.226068 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 19476
I20260812 01:03:08.226989 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:35975
--local_ip_for_outbound_sockets=127.21.201.67
--tserver_master_addrs=127.21.201.126:32907
--webserver_port=34809
--webserver_interface=127.21.201.67
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:39133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:03:08.228536 19699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:08.225902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.66:46859)
0812 01:03:08.226844 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:08.227169 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":361,"threads_started":1}
W20260812 01:03:08.229454 19696 heartbeater.cc:644] Failed to heartbeat to 127.21.201.126:32907 (0 consecutive failures): Network error: Failed to ping master at 127.21.201.126:32907: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
I20260812 01:03:08.230682 19699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:08.230176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.66:36735)
0812 01:03:08.230398 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:08.230515 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260812 01:03:08.231907 19699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:08.231409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.66:57239)
0812 01:03:08.231601 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:08.231709 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260812 01:03:08.232214 19696 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 01:03:08.575453 19701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:08.575788 19701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:08.575902 19701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:08.583452 19701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:08.583581 19701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:08.583702 19701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 01:03:08.593863 19701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:39133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:35975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=34809
--tserver_master_addrs=127.21.201.126:32907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:08.595171 19701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:08.596565 19701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:08.610342 19707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:08.610379 19712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:08.610299 19708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:08.611375 19701 server_base.cc:1034] running on GCE node
I20260812 01:03:08.612193 19701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:08.613269 19701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:08.614492 19701 hybrid_clock.cc:648] HybridClock initialized: now 1786496588614442 us; error 42 us; skew 500 ppm
I20260812 01:03:08.614879 19701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:03:08 dist-test-slave-7f01 krb5kdc[17209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496588, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:03:08.618526 19701 init.cc:377] Logged in from keytab as kudu/127.21.201.67@KRBTEST.COM (short username kudu)
I20260812 01:03:08.620306 19701 webserver.cc:460] Webserver started at http://127.21.201.67:34809/ using document root <none> and password file <none>
I20260812 01:03:08.621054 19701 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:08.621214 19701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:08.626973 19701 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260812 01:03:08.630309 19725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:08.632251 19701 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 01:03:08.632462 19701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "762b216bab804e69ae171e5ae21e1bd1"
format_stamp: "Formatted at 2026-08-12 01:02:22 on dist-test-slave-7f01"
server_key: "508e56cdcb206f2d1a87853429c21f0b"
server_key_iv: "92f0e3a670001f51d1d3acf730a87bd0"
server_key_version: "encryptionkey@0"
I20260812 01:03:08.633577 19701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:08.653156 19701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:08.658003 19701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:08.658334 19701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:08.659467 19701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:03:08.660964 19701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:03:08.661058 19701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:08.661192 19701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:03:08.661293 19701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:08.698611 19701 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:35975
I20260812 01:03:08.698647 19920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:35975 every 8 connection(s)
I20260812 01:03:08.700539 19701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 01:03:08.705577 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:08.703594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.67:48727)
0812 01:03:08.704474 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:08.704710 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":328,"threads_started":1}
W20260812 01:03:08.706348 19921 heartbeater.cc:644] Failed to heartbeat to 127.21.201.126:32907 (0 consecutive failures): Network error: Failed to ping master at 127.21.201.126:32907: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
I20260812 01:03:08.707300 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:08.706977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.67:41645)
0812 01:03:08.707111 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:08.707186 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 01:03:08.707315 19928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:08.706517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.65:40823)
0812 01:03:08.706984 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:08.707085 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":253,"threads_started":1}
I20260812 01:03:08.708325 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:08.707965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.67:57145)
0812 01:03:08.708131 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:08.708213 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260812 01:03:08.708606 19921 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 01:03:08.710790 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 19701
I20260812 01:03:09.233705 19931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:09.232814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.66:49407)
0812 01:03:09.233346 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:09.233442 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":271,"threads_started":1}
I20260812 01:03:09.709038 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:09.708249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.65:59453)
0812 01:03:09.708712 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:09.708800 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":226,"threads_started":1}
I20260812 01:03:09.710028 19933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:09.709206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.67:48323)
0812 01:03:09.709706 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:09.709809 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":269,"threads_started":1}
I20260812 01:03:10.235941 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:10.234956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.66:35865)
0812 01:03:10.235583 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:10.235703 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":285,"threads_started":1}
I20260812 01:03:10.711215 19938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:10.710169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.65:52175)
0812 01:03:10.710814 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:10.710940 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":306,"threads_started":1}
I20260812 01:03:10.711895 19939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:10.710999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.67:55397)
0812 01:03:10.711554 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:10.711663 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":229,"threads_started":1}
I20260812 01:03:11.237816 19941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:11.237030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.66:41185)
0812 01:03:11.237482 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:11.237570 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":232,"threads_started":1}
I20260812 01:03:11.713162 19942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:11.712305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.65:38087)
0812 01:03:11.712883 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:11.712978 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":323,"threads_started":1}
I20260812 01:03:11.713780 19950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:11.712898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.67:38811)
0812 01:03:11.713532 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:11.713616 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":287,"threads_started":1}
I20260812 01:03:12.239877 19951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:12.239000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.66:36559)
0812 01:03:12.239519 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:12.239608 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":273,"threads_started":1}
I20260812 01:03:12.715005 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:12.714163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.65:42109)
0812 01:03:12.714739 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:12.714824 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":322,"threads_started":1}
I20260812 01:03:12.715502 19958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:12.714638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.67:60443)
0812 01:03:12.715218 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:12.715326 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":327,"threads_started":1}
I20260812 01:03:13.241813 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:13.240919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.66:52391)
0812 01:03:13.241488 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:13.241576 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":326,"threads_started":1}
I20260812 01:03:13.717232 19961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:13.716328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.65:38649)
0812 01:03:13.716894 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:13.717015 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":259,"threads_started":1}
I20260812 01:03:13.717453 19962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:13.716510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.67:41689)
0812 01:03:13.717107 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:13.717218 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":294,"threads_started":1}
I20260812 01:03:14.243634 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:14.242863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.66:42817)
0812 01:03:14.243359 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:14.243455 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":254,"threads_started":1}
I20260812 01:03:14.719236 19969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:14.718371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.67:52949)
0812 01:03:14.718970 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:14.719075 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":318,"threads_started":1}
I20260812 01:03:14.719234 19968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:14.718256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.65:32883)
0812 01:03:14.718924 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:14.719017 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":359,"threads_started":1}
I20260812 01:03:15.245632 19970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:15.244688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.66:34783)
0812 01:03:15.245310 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:15.245403 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":328,"threads_started":1}
I20260812 01:03:15.721222 19971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:15.720332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.67:54841)
0812 01:03:15.720865 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:15.720954 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":236,"threads_started":1}
I20260812 01:03:15.721632 19972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:15.720332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.65:41875)
0812 01:03:15.721191 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:15.721285 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":640,"threads_started":1}
I20260812 01:03:16.247471 19973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:16.246520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.66:36347)
0812 01:03:16.247194 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:16.247283 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":350,"threads_started":1}
I20260812 01:03:16.723145 19974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:16.722272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.67:52451)
0812 01:03:16.722783 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:16.722912 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":265,"threads_started":1}
I20260812 01:03:16.723699 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:16.722541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.65:52149)
0812 01:03:16.723206 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:16.723475 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":369,"threads_started":1}
I20260812 01:03:17.249210 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:17.248329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.66:48973)
0812 01:03:17.248803 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:17.248974 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":231,"threads_started":1}
I20260812 01:03:17.725032 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:17.724234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.67:48319)
0812 01:03:17.724756 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:17.724841 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":284,"threads_started":1}
I20260812 01:03:17.725600 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:17.724709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.65:36835)
0812 01:03:17.725360 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:17.725445 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":278,"threads_started":1}
I20260812 01:03:18.251045 19987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:18.250194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.66:52693)
0812 01:03:18.250695 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:18.250814 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":279,"threads_started":1}
I20260812 01:03:18.727042 19988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:18.726088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.67:47673)
0812 01:03:18.726645 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:18.726808 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":272,"threads_started":1}
I20260812 01:03:18.727411 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:18.726519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.65:57501)
0812 01:03:18.727052 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:18.727172 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":322,"threads_started":1}
I20260812 01:03:19.252862 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:19.252079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.66:56795)
0812 01:03:19.252559 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:19.252661 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":233,"threads_started":1}
I20260812 01:03:19.729087 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:19.728100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.67:43555)
0812 01:03:19.728735 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:19.728864 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":300,"threads_started":1}
I20260812 01:03:19.729166 19992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:19.728442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.65:55671)
0812 01:03:19.728917 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:19.729001 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":258,"threads_started":1}
I20260812 01:03:20.254571 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:20.253862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.66:33911)
0812 01:03:20.254313 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:20.254400 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":228,"threads_started":1}
I20260812 01:03:20.731108 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:20.730257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.67:56007)
0812 01:03:20.730831 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:20.730925 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":311,"threads_started":1}
I20260812 01:03:20.731166 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:20.730257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.65:47613)
0812 01:03:20.730855 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:20.730965 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":308,"threads_started":1}
I20260812 01:03:21.256559 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:21.255650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.66:36281)
0812 01:03:21.256217 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:21.256337 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":307,"threads_started":1}
I20260812 01:03:21.733198 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:21.732195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.67:37291)
0812 01:03:21.732844 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:21.732999 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":373,"threads_started":1}
I20260812 01:03:21.733305 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:21.732209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.65:53419)
0812 01:03:21.732834 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:21.733058 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":328,"threads_started":1}
I20260812 01:03:22.258297 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:22.257527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.66:41269)
0812 01:03:22.257996 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:22.258086 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":244,"threads_started":1}
I20260812 01:03:22.735316 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:22.734291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.67:46817)
0812 01:03:22.734887 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:22.735010 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":262,"threads_started":1}
I20260812 01:03:22.735811 20003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:22.734866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.65:49283)
0812 01:03:22.735378 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:22.735549 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":271,"threads_started":1}
I20260812 01:03:23.260325 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:23.259470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.66:47789)
0812 01:03:23.259998 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:23.260116 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":294,"threads_started":1}
I20260812 01:03:23.718209 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 19171
I20260812 01:03:23.738173 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:23.736657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.67:41115)
0812 01:03:23.737674 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:23.737882 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":882,"thread_start_us":770,"threads_started":1}
I20260812 01:03:23.738706 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:23.736977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:32907 (local address 127.21.201.65:37419)
0812 01:03:23.738174 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:23.738421 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.126:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1046,"thread_start_us":830,"threads_started":1}
I20260812 01:03:23.741677 20005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:23.719471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36432 (local address 127.21.201.65:41747)
0812 01:03:23.720161 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:23.720180 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:23.720235 (+ 55us) server_negotiation.cc:407] Connection header received
0812 01:03:23.722288 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:23.722300 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:23.722708 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:23.723010 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:23.724541 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:23.725827 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:23.727194 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:23.728021 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:23.728169 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:23.733935 (+ 5766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:23.733944 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:23.733958 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:23.733997 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:23.738156 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:23.739008 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:23.739017 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:23.739026 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:23.739219 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:23.739594 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:23.739597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:23.739599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:23.740217 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:23.740459 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:23.740897 (+ 438us) server_negotiation.cc:299] Negotiation successful
0812 01:03:23.741345 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":339,"threads_started":1}
I20260812 01:03:23.743304 19466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:23.832772 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 19171
I20260812 01:03:23.860708 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 19476
I20260812 01:03:23.880703 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:23.861778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47798 (local address 127.21.201.66:40855)
0812 01:03:23.862340 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:23.862354 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:23.862409 (+ 55us) server_negotiation.cc:407] Connection header received
0812 01:03:23.864210 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:23.864221 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:23.864654 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:23.864943 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:23.866462 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:23.867921 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:23.869177 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:23.869835 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:23.869974 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:23.872741 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:23.872748 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:23.872763 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:23.872801 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:23.875610 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:23.876486 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:23.876500 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:23.876518 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:23.876797 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:23.877357 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:23.877363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:23.877367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:23.878208 (+ 841us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:23.878631 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:23.879648 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0812 01:03:23.880175 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":270,"threads_started":1}
I20260812 01:03:23.883188 19689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:23.983006 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 19476
I20260812 01:03:24.010223 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 19701
I20260812 01:03:24.029047 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:24.011311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52462 (local address 127.21.201.67:35975)
0812 01:03:24.012023 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:24.012038 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:24.012084 (+ 46us) server_negotiation.cc:407] Connection header received
0812 01:03:24.013809 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:24.013819 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:24.014232 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:24.014593 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:24.016370 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:24.017581 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:24.018823 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:24.019488 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:24.019702 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:24.022474 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:24.022481 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:24.022495 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:24.022530 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:24.025527 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:24.026279 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:24.026286 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:24.026292 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:24.026469 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:24.026881 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:24.026887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:24.026890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:24.027481 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:24.027839 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:24.028273 (+ 434us) server_negotiation.cc:299] Negotiation successful
0812 01:03:24.028717 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":322,"threads_started":1}
I20260812 01:03:24.030381 19914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:24.116385 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 19701
2026-08-12T01:03:24Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65634 ms)
[----------] 30 tests from SecurityITest (325307 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:03:24 dist-test-slave-7f01 krb5kdc[20022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:03:24 dist-test-slave-7f01 krb5kdc[20022](info): set up 2 sockets
Aug 12 01:03:24 dist-test-slave-7f01 krb5kdc[20022](info): commencing operation
krb5kdc: starting...
W20260812 01:03:25.531116 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.368s user 0.002s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:03:25 dist-test-slave-7f01 krb5kdc[20022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:03:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:03:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 01:03:25.843402 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:38499
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:38499
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:03:26.201696 20049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:26.201987 20049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:26.202159 20049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:26.209383 20049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:03:26.209525 20049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:26.209614 20049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:26.209724 20049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:03:26.209806 20049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:03:26.219507 20049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:38499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:38499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:26.220912 20049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:26.222257 20049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:26.234841 20057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:26.235046 20056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:26.235054 20061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:26.236034 20049 server_base.cc:1034] running on GCE node
I20260812 01:03:26.236771 20049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:26.238189 20049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:26.239415 20049 hybrid_clock.cc:648] HybridClock initialized: now 1786496606239360 us; error 34 us; skew 500 ppm
I20260812 01:03:26.239866 20049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:03:26 dist-test-slave-7f01 krb5kdc[20022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496606, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:03:26.243333 20049 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 01:03:26.244938 20049 webserver.cc:460] Webserver started at http://127.21.201.126:42449/ using document root <none> and password file <none>
I20260812 01:03:26.245709 20049 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:26.245873 20049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:26.246186 20049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:26.249409 20049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "9e26c18bf67e4bef80964d1cba273c16"
format_stamp: "Formatted at 2026-08-12 01:03:26 on dist-test-slave-7f01"
server_key: "ede0c27e34996fd8e5df4e79e4e5052a"
server_key_iv: "51a25c235f14f166a5a20b8561b62ed3"
server_key_version: "encryptionkey@0"
I20260812 01:03:26.250190 20049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "9e26c18bf67e4bef80964d1cba273c16"
format_stamp: "Formatted at 2026-08-12 01:03:26 on dist-test-slave-7f01"
server_key: "ede0c27e34996fd8e5df4e79e4e5052a"
server_key_iv: "51a25c235f14f166a5a20b8561b62ed3"
server_key_version: "encryptionkey@0"
I20260812 01:03:26.255463 20049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 01:03:26.259068 20069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:26.260603 20049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:03:26.260809 20049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "9e26c18bf67e4bef80964d1cba273c16"
format_stamp: "Formatted at 2026-08-12 01:03:26 on dist-test-slave-7f01"
server_key: "ede0c27e34996fd8e5df4e79e4e5052a"
server_key_iv: "51a25c235f14f166a5a20b8561b62ed3"
server_key_version: "encryptionkey@0"
I20260812 01:03:26.261032 20049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:26.274775 20049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:26.278929 20049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:26.279232 20049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:26.299237 20049 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:38499
I20260812 01:03:26.299259 20152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:38499 every 8 connection(s)
I20260812 01:03:26.300688 20049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:03:26.302436 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 20049
I20260812 01:03:26.302685 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:03:26.303596 22309 external_mini_cluster.cc:1428] Setting key c7cae8541eb345f2cff56453cecf2f00
I20260812 01:03:26.305569 20153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 01:03:26 dist-test-slave-7f01 krb5kdc[20022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:03:26.319602 20153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e26c18bf67e4bef80964d1cba273c16: Bootstrap starting.
I20260812 01:03:26.323945 20153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e26c18bf67e4bef80964d1cba273c16: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:26.325254 20153 log.cc:824] T 00000000000000000000000000000000 P 9e26c18bf67e4bef80964d1cba273c16: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:26.328841 20153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e26c18bf67e4bef80964d1cba273c16: No bootstrap required, opened a new log
I20260812 01:03:26.329384 20157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:26.305846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44700 (local address 127.21.201.126:38499)
0812 01:03:26.306680 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:26.306695 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:26.306748 (+ 53us) server_negotiation.cc:407] Connection header received
0812 01:03:26.308988 (+ 2240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:26.309024 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:26.309523 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:26.309967 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:26.311408 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:26.313546 (+ 2138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:26.314896 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:26.315965 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:26.316176 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:26.320513 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:26.320524 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:26.320542 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:26.320592 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:26.324705 (+ 4113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:26.325477 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:26.325485 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:26.325492 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:26.325722 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:26.326231 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:26.326234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:26.326236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:26.326892 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:26.327176 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:26.327574 (+ 398us) server_negotiation.cc:299] Negotiation successful
0812 01:03:26.328058 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":342,"threads_started":1}
I20260812 01:03:26.337148 20153 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e26c18bf67e4bef80964d1cba273c16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e26c18bf67e4bef80964d1cba273c16" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38499 } }
I20260812 01:03:26.337502 20153 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e26c18bf67e4bef80964d1cba273c16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:26.337597 20153 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e26c18bf67e4bef80964d1cba273c16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e26c18bf67e4bef80964d1cba273c16, State: Initialized, Role: FOLLOWER
I20260812 01:03:26.338068 20153 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e26c18bf67e4bef80964d1cba273c16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e26c18bf67e4bef80964d1cba273c16" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38499 } }
I20260812 01:03:26.338402 20153 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e26c18bf67e4bef80964d1cba273c16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:03:26.338580 20153 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e26c18bf67e4bef80964d1cba273c16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:03:26.338778 20153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e26c18bf67e4bef80964d1cba273c16 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:26.341073 20153 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e26c18bf67e4bef80964d1cba273c16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e26c18bf67e4bef80964d1cba273c16" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38499 } }
I20260812 01:03:26.341606 20153 leader_election.cc:302] T 00000000000000000000000000000000 P 9e26c18bf67e4bef80964d1cba273c16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e26c18bf67e4bef80964d1cba273c16; no voters:
I20260812 01:03:26.342234 20153 leader_election.cc:288] T 00000000000000000000000000000000 P 9e26c18bf67e4bef80964d1cba273c16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:03:26.342393 20159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e26c18bf67e4bef80964d1cba273c16 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:03:26.343477 20159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e26c18bf67e4bef80964d1cba273c16 [term 1 LEADER]: Becoming Leader. State: Replica: 9e26c18bf67e4bef80964d1cba273c16, State: Running, Role: LEADER
I20260812 01:03:26.344049 20159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e26c18bf67e4bef80964d1cba273c16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e26c18bf67e4bef80964d1cba273c16" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38499 } }
I20260812 01:03:26.344653 20153 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e26c18bf67e4bef80964d1cba273c16 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:03:26.346822 20162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e26c18bf67e4bef80964d1cba273c16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e26c18bf67e4bef80964d1cba273c16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e26c18bf67e4bef80964d1cba273c16" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38499 } } }
I20260812 01:03:26.347196 20162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e26c18bf67e4bef80964d1cba273c16 [sys.catalog]: This master's current role is: LEADER
I20260812 01:03:26.348143 20167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:03:26.347992 20163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e26c18bf67e4bef80964d1cba273c16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e26c18bf67e4bef80964d1cba273c16. Latest consensus state: current_term: 1 leader_uuid: "9e26c18bf67e4bef80964d1cba273c16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e26c18bf67e4bef80964d1cba273c16" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 38499 } } }
I20260812 01:03:26.348376 20163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e26c18bf67e4bef80964d1cba273c16 [sys.catalog]: This master's current role is: LEADER
I20260812 01:03:26.354128 20167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:03:26.364145 20167 catalog_manager.cc:1347] Generated new cluster ID: b81a913f413c485ebb377265ccbab2f3
I20260812 01:03:26.364289 20167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:03:26.378219 20167 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:03:26.379536 20167 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:03:26.387712 20167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e26c18bf67e4bef80964d1cba273c16: Generated new TSK 0
I20260812 01:03:26.388538 20167 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.201.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
I20260812 01:03:26.473721 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:0
--local_ip_for_outbound_sockets=127.21.201.65
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38499
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:03:26.835019 20188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:26.835340 20188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:26.835443 20188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:26.842797 20188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:26.842911 20188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:26.843031 20188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 01:03:26.853526 20188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:26.854851 20188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:26.856266 20188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:26.869966 20198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:26.869906 20202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:26.869874 20200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:26.870802 20188 server_base.cc:1034] running on GCE node
I20260812 01:03:26.871851 20188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:26.872887 20188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:26.874111 20188 hybrid_clock.cc:648] HybridClock initialized: now 1786496606874053 us; error 55 us; skew 500 ppm
I20260812 01:03:26.874402 20188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:03:26 dist-test-slave-7f01 krb5kdc[20022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496606, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:03:26.878034 20188 init.cc:377] Logged in from keytab as kudu/127.21.201.65@KRBTEST.COM (short username kudu)
I20260812 01:03:26.879787 20188 webserver.cc:460] Webserver started at http://127.21.201.65:42185/ using document root <none> and password file <none>
I20260812 01:03:26.880492 20188 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:26.880605 20188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:26.880872 20188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:26.883898 20188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-0/data/instance:
uuid: "6bc61720a6064cb390db4072579ab595"
format_stamp: "Formatted at 2026-08-12 01:03:26 on dist-test-slave-7f01"
server_key: "9e4f583061b27664e682f14643894a05"
server_key_iv: "79d31f08bbe2fe9647ce569ee85b7704"
server_key_version: "encryptionkey@0"
I20260812 01:03:26.884667 20188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance:
uuid: "6bc61720a6064cb390db4072579ab595"
format_stamp: "Formatted at 2026-08-12 01:03:26 on dist-test-slave-7f01"
server_key: "9e4f583061b27664e682f14643894a05"
server_key_iv: "79d31f08bbe2fe9647ce569ee85b7704"
server_key_version: "encryptionkey@0"
I20260812 01:03:26.889891 20188 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:03:26.893507 20216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:26.894896 20188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 01:03:26.895028 20188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "6bc61720a6064cb390db4072579ab595"
format_stamp: "Formatted at 2026-08-12 01:03:26 on dist-test-slave-7f01"
server_key: "9e4f583061b27664e682f14643894a05"
server_key_iv: "79d31f08bbe2fe9647ce569ee85b7704"
server_key_version: "encryptionkey@0"
I20260812 01:03:26.895193 20188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:26.908572 20188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:26.912976 20188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:26.913230 20188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:26.914124 20188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:03:26.915431 20188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:03:26.915508 20188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:26.915593 20188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:03:26.915643 20188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:26.950763 20188 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:39589
I20260812 01:03:26.950877 20405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:39589 every 8 connection(s)
I20260812 01:03:26.952229 20188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 01:03:26.959666 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 20188
I20260812 01:03:26.959955 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance
I20260812 01:03:26.960638 22309 external_mini_cluster.cc:1428] Setting key b465721a4b985c4ecca8db6c69a3602f
Aug 12 01:03:26 dist-test-slave-7f01 krb5kdc[20022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496606, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:03:26.975849 20408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:26.955153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:53033 (local address 127.21.201.126:38499)
0812 01:03:26.955627 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:26.955632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:26.956722 (+ 1090us) server_negotiation.cc:407] Connection header received
0812 01:03:26.958685 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:26.958692 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:26.958793 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:26.958919 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:26.960651 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:26.961452 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:26.963370 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:26.963862 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:26.963995 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:26.968149 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:26.968158 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:26.968166 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:26.968214 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:26.972225 (+ 4011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:26.973025 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:26.973034 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:26.973040 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:26.973203 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:26.973695 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:26.973702 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:26.973708 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:26.974122 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:26.974301 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:26.975026 (+ 725us) server_negotiation.cc:299] Negotiation successful
0812 01:03:26.975308 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":228,"threads_started":1}
I20260812 01:03:26.976665 20409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:26.955789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38499 (local address 127.21.201.65:53033)
0812 01:03:26.956558 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:26.956602 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:26.958378 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:26.959169 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:26.959177 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:26.959813 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:03:26.960471 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:26.960489 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:26.961651 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:26.961677 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:26.963217 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:26.963230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:26.964122 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:26.964130 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:26.964359 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:26.965143 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:03:26.965165 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:03:26.967900 (+ 2735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:03:26.972400 (+ 4500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:03:26.972408 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:03:26.972421 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:03:26.972880 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:03:26.973344 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:03:26.973348 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:03:26.973351 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:03:26.973566 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:03:26.974264 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:03:26.974270 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:03:26.974612 (+ 342us) client_negotiation.cc:769] Sending connection context
0812 01:03:26.974846 (+ 234us) client_negotiation.cc:240] Negotiation successful
0812 01:03:26.975277 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":284,"threads_started":1}
I20260812 01:03:26.978361 20406 heartbeater.cc:347] Connected to a master server at 127.21.201.126:38499
I20260812 01:03:26.978675 20406 heartbeater.cc:464] Registering TS with master...
I20260812 01:03:26.979444 20406 heartbeater.cc:511] Master 127.21.201.126:38499 requested a full tablet report, sending...
I20260812 01:03:26.981678 20096 ts_manager.cc:194] Registered new tserver with Master: 6bc61720a6064cb390db4072579ab595 (127.21.201.65:39589)
I20260812 01:03:26.983175 20096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.65@KRBTEST.COM'} at 127.21.201.65:53033
WARNING: no policy specified for kudu/127.21.201.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
I20260812 01:03:27.031524 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:0
--local_ip_for_outbound_sockets=127.21.201.66
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38499
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:03:27.380100 20418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:27.380426 20418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:27.380601 20418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:27.388083 20418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:27.388242 20418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:27.388437 20418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 01:03:27.398191 20418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:27.399611 20418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:27.401075 20418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:27.414320 20423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:27.414345 20428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:27.414285 20424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:27.415284 20418 server_base.cc:1034] running on GCE node
I20260812 01:03:27.416013 20418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:27.417009 20418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:27.418241 20418 hybrid_clock.cc:648] HybridClock initialized: now 1786496607418184 us; error 53 us; skew 500 ppm
I20260812 01:03:27.418557 20418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:03:27 dist-test-slave-7f01 krb5kdc[20022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496607, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:03:27.421983 20418 init.cc:377] Logged in from keytab as kudu/127.21.201.66@KRBTEST.COM (short username kudu)
I20260812 01:03:27.423497 20418 webserver.cc:460] Webserver started at http://127.21.201.66:37863/ using document root <none> and password file <none>
I20260812 01:03:27.424234 20418 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:27.424386 20418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:27.424710 20418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:27.427601 20418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-1/data/instance:
uuid: "7de1baf8631f475d8a984ba7afffe1b6"
format_stamp: "Formatted at 2026-08-12 01:03:27 on dist-test-slave-7f01"
server_key: "1c7dbc659e3d6096143685b5118a5e13"
server_key_iv: "7bbb8f6a602391359512a3d100626eac"
server_key_version: "encryptionkey@0"
I20260812 01:03:27.428459 20418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance:
uuid: "7de1baf8631f475d8a984ba7afffe1b6"
format_stamp: "Formatted at 2026-08-12 01:03:27 on dist-test-slave-7f01"
server_key: "1c7dbc659e3d6096143685b5118a5e13"
server_key_iv: "7bbb8f6a602391359512a3d100626eac"
server_key_version: "encryptionkey@0"
I20260812 01:03:27.433698 20418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:03:27.437214 20438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:27.438530 20418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:03:27.438719 20418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "7de1baf8631f475d8a984ba7afffe1b6"
format_stamp: "Formatted at 2026-08-12 01:03:27 on dist-test-slave-7f01"
server_key: "1c7dbc659e3d6096143685b5118a5e13"
server_key_iv: "7bbb8f6a602391359512a3d100626eac"
server_key_version: "encryptionkey@0"
I20260812 01:03:27.438941 20418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:27.452771 20418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:27.456816 20418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:27.457144 20418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:27.458134 20418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:03:27.459472 20418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:03:27.459580 20418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:27.459777 20418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:03:27.459863 20418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260812 01:03:27.496351 20418 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:33495
I20260812 01:03:27.496372 20630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:33495 every 8 connection(s)
I20260812 01:03:27.497750 20418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 01:03:27.502620 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 20418
I20260812 01:03:27.502848 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance
I20260812 01:03:27.503487 22309 external_mini_cluster.cc:1428] Setting key 3657964fb4174abc3e1caf9f3ba07439
Aug 12 01:03:27 dist-test-slave-7f01 krb5kdc[20022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496607, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:03:27.521836 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:27.500922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:44341 (local address 127.21.201.126:38499)
0812 01:03:27.501448 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:27.501455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:27.505365 (+ 3910us) server_negotiation.cc:407] Connection header received
0812 01:03:27.505458 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:27.505466 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:27.505555 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:27.505654 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:27.507297 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:27.508565 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:27.510116 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:27.510882 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:27.511029 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:27.514582 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:27.514591 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:27.514598 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:27.514644 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:27.518398 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:27.519127 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:27.519135 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:27.519140 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:27.519320 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:27.519783 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:27.519790 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:27.519794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:27.520198 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:27.520383 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:27.521036 (+ 653us) server_negotiation.cc:299] Negotiation successful
0812 01:03:27.521309 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":252,"threads_started":1}
I20260812 01:03:27.522465 20635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:27.501577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38499 (local address 127.21.201.66:44341)
0812 01:03:27.502644 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:27.502699 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:27.505085 (+ 2386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:27.505954 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:27.505962 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:27.506497 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:03:27.507098 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:27.507117 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:27.508717 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:27.508720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:27.509954 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:27.509966 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:27.511097 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:27.511102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:27.511294 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:27.512107 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:03:27.512131 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:03:27.514338 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:03:27.518551 (+ 4213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:03:27.518558 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:03:27.518571 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:03:27.518993 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:03:27.519447 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:03:27.519450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:03:27.519452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:03:27.519641 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:03:27.520328 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:03:27.520334 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:03:27.520665 (+ 331us) client_negotiation.cc:769] Sending connection context
0812 01:03:27.520900 (+ 235us) client_negotiation.cc:240] Negotiation successful
0812 01:03:27.521290 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":498,"threads_started":1}
I20260812 01:03:27.524155 20631 heartbeater.cc:347] Connected to a master server at 127.21.201.126:38499
I20260812 01:03:27.524530 20631 heartbeater.cc:464] Registering TS with master...
I20260812 01:03:27.525454 20631 heartbeater.cc:511] Master 127.21.201.126:38499 requested a full tablet report, sending...
I20260812 01:03:27.527298 20096 ts_manager.cc:194] Registered new tserver with Master: 7de1baf8631f475d8a984ba7afffe1b6 (127.21.201.66:33495)
I20260812 01:03:27.528241 20096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.66@KRBTEST.COM'} at 127.21.201.66:44341
WARNING: no policy specified for kudu/127.21.201.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
I20260812 01:03:27.575980 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:0
--local_ip_for_outbound_sockets=127.21.201.67
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38499
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:03:27.929497 20638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:27.929787 20638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:27.929889 20638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:27.937552 20638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:27.937670 20638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:27.937790 20638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 01:03:27.948163 20638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:38499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:27.949501 20638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:27.950977 20638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:27.964428 20648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:27.964509 20650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:27.964480 20655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:27.966064 20638 server_base.cc:1034] running on GCE node
I20260812 01:03:27.966733 20638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:27.967831 20638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:27.969074 20638 hybrid_clock.cc:648] HybridClock initialized: now 1786496607969020 us; error 47 us; skew 500 ppm
I20260812 01:03:27.969406 20638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:03:27 dist-test-slave-7f01 krb5kdc[20022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496607, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:03:27.972707 20638 init.cc:377] Logged in from keytab as kudu/127.21.201.67@KRBTEST.COM (short username kudu)
I20260812 01:03:27.974232 20638 webserver.cc:460] Webserver started at http://127.21.201.67:40961/ using document root <none> and password file <none>
I20260812 01:03:27.974939 20638 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:27.975106 20638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:27.975433 20638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:27.978475 20638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-2/data/instance:
uuid: "ceb49de3cb994c55a33a2c5bc86a591b"
format_stamp: "Formatted at 2026-08-12 01:03:27 on dist-test-slave-7f01"
server_key: "5a9b40d7a6ce1e27f3c740f731bb5f70"
server_key_iv: "250328122be0c76cc71447dad2a97d8f"
server_key_version: "encryptionkey@0"
I20260812 01:03:27.979301 20638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance:
uuid: "ceb49de3cb994c55a33a2c5bc86a591b"
format_stamp: "Formatted at 2026-08-12 01:03:27 on dist-test-slave-7f01"
server_key: "5a9b40d7a6ce1e27f3c740f731bb5f70"
server_key_iv: "250328122be0c76cc71447dad2a97d8f"
server_key_version: "encryptionkey@0"
I20260812 01:03:27.984697 20638 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 01:03:27.986251 20406 heartbeater.cc:503] Master 127.21.201.126:38499 was elected leader, sending a full tablet report...
I20260812 01:03:27.988660 20663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:27.990025 20638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:03:27.990164 20638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "ceb49de3cb994c55a33a2c5bc86a591b"
format_stamp: "Formatted at 2026-08-12 01:03:27 on dist-test-slave-7f01"
server_key: "5a9b40d7a6ce1e27f3c740f731bb5f70"
server_key_iv: "250328122be0c76cc71447dad2a97d8f"
server_key_version: "encryptionkey@0"
I20260812 01:03:27.990331 20638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:28.030826 20638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:28.034968 20638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:28.035228 20638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:28.036226 20638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:03:28.037575 20638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:03:28.037650 20638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:28.037738 20638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:03:28.037784 20638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:28.073366 20638 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:34405
I20260812 01:03:28.073467 20849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:34405 every 8 connection(s)
I20260812 01:03:28.074723 20638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 01:03:28.077009 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 20638
I20260812 01:03:28.077208 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance
I20260812 01:03:28.077865 22309 external_mini_cluster.cc:1428] Setting key 70b16afd8ce4340dd9ed6add1b91755a
Aug 12 01:03:28 dist-test-slave-7f01 krb5kdc[20022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496607, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:03:28.096984 20854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:28.077530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:58377 (local address 127.21.201.126:38499)
0812 01:03:28.078120 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:28.078128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:28.079191 (+ 1063us) server_negotiation.cc:407] Connection header received
0812 01:03:28.081724 (+ 2533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:28.081728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:28.081797 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:28.081888 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:28.083475 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.084351 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.085939 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.086325 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.086431 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:28.090001 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:28.090006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:28.090009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:28.090041 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:28.093321 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:28.094347 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:28.094352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:28.094354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:28.094530 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:28.095146 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:28.095153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:28.095159 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:28.095587 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:28.095796 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:28.096472 (+ 676us) server_negotiation.cc:299] Negotiation successful
0812 01:03:28.096650 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":261,"threads_started":1}
I20260812 01:03:28.098069 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:28.078136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:38499 (local address 127.21.201.67:58377)
0812 01:03:28.079001 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:28.079062 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:28.081477 (+ 2415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:28.082165 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:28.082173 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:28.082709 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:03:28.083295 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:28.083314 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.084508 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.084512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:28.085778 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:28.085791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.086531 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.086536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:28.086729 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:28.087485 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:03:28.087508 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:03:28.089807 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:03:28.093495 (+ 3688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:03:28.093501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:03:28.093514 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:03:28.094226 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:03:28.094659 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:03:28.094665 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:03:28.094669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:03:28.094971 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:03:28.095772 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:03:28.095781 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:03:28.096134 (+ 353us) client_negotiation.cc:769] Sending connection context
0812 01:03:28.096383 (+ 249us) client_negotiation.cc:240] Negotiation successful
0812 01:03:28.096806 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":332,"threads_started":1}
I20260812 01:03:28.099395 20850 heartbeater.cc:347] Connected to a master server at 127.21.201.126:38499
I20260812 01:03:28.099869 20850 heartbeater.cc:464] Registering TS with master...
I20260812 01:03:28.100646 20850 heartbeater.cc:511] Master 127.21.201.126:38499 requested a full tablet report, sending...
I20260812 01:03:28.102093 20096 ts_manager.cc:194] Registered new tserver with Master: ceb49de3cb994c55a33a2c5bc86a591b (127.21.201.67:34405)
I20260812 01:03:28.102974 20096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.67@KRBTEST.COM'} at 127.21.201.67:58377
I20260812 01:03:28.106315 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:03:28.124426 20854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:28.111125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44150 (local address 127.21.201.126:38499)
0812 01:03:28.111297 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:28.111302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:28.111820 (+ 518us) server_negotiation.cc:407] Connection header received
0812 01:03:28.112000 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:28.112004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:28.112069 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:28.112154 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:28.113921 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.114690 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.116168 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.116576 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.116668 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:28.119150 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:28.119155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:28.119159 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:28.119192 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:28.121862 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:28.122483 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:28.122487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:28.122489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:28.122627 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:28.122962 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:28.122965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:28.122967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:28.123305 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:28.123492 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:28.123962 (+ 470us) server_negotiation.cc:299] Negotiation successful
0812 01:03:28.124156 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 01:03:28.128834 20096 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:03:28.136016 20096 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:03:28.172314 20870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:28.149509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:39589 (local address 127.0.0.1:54708)
0812 01:03:28.163608 (+ 14099us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:28.163632 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:28.164299 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:28.165065 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:28.165074 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:28.165107 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:28.165634 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:28.165661 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.167538 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.167543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:28.169551 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:28.169564 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.171283 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.171290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:28.171496 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:28.171547 (+ 51us) client_negotiation.cc:769] Sending connection context
0812 01:03:28.171682 (+ 135us) client_negotiation.cc:240] Negotiation successful
0812 01:03:28.171899 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13942,"thread_start_us":12025,"threads_started":1}
I20260812 01:03:28.173177 20875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:28.149672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54708 (local address 127.21.201.65:39589)
0812 01:03:28.163325 (+ 13653us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:28.163337 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:28.164106 (+ 769us) server_negotiation.cc:407] Connection header received
0812 01:03:28.164528 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:28.164536 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:28.164814 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:28.165702 (+ 888us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:28.166002 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.167362 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.169723 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.170707 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.170878 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:28.171133 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:28.172596 (+ 1463us) server_negotiation.cc:299] Negotiation successful
0812 01:03:28.172795 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13424,"thread_start_us":13201,"threads_started":1}
I20260812 01:03:28.173624 20878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:28.163009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42060 (local address 127.21.201.67:34405)
0812 01:03:28.163783 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:28.163790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:28.164567 (+ 777us) server_negotiation.cc:407] Connection header received
0812 01:03:28.164869 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:28.164877 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:28.165105 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:28.165291 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:28.166535 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.168191 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.170830 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.172203 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.172357 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:28.172605 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:28.172950 (+ 345us) server_negotiation.cc:299] Negotiation successful
0812 01:03:28.173141 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":401,"threads_started":1}
I20260812 01:03:28.173694 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:28.149178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:33495 (local address 127.0.0.1:38066)
0812 01:03:28.162124 (+ 12946us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:28.162186 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:28.162453 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:28.165331 (+ 2878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:28.165336 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:28.165357 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:28.166044 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:28.166055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.168022 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.168028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:28.170567 (+ 2539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:28.170588 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.172825 (+ 2237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.172831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:28.173018 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:28.173049 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 01:03:28.173127 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 01:03:28.173270 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12543,"mutex_wait_us":80,"thread_start_us":14564,"threads_started":1}
I20260812 01:03:28.174587 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:28.149414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38066 (local address 127.21.201.66:33495)
0812 01:03:28.164820 (+ 15406us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:28.164828 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:28.164855 (+ 27us) server_negotiation.cc:407] Connection header received
0812 01:03:28.164945 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:28.164952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:28.165131 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:28.165311 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:28.166209 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.167857 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.170760 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.172230 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.172373 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:28.174001 (+ 1628us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:28.174072 (+ 71us) server_negotiation.cc:299] Negotiation successful
0812 01:03:28.174241 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15207,"thread_start_us":15568,"threads_started":1}
I20260812 01:03:28.174314 20880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:28.163750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:34405 (local address 127.0.0.1:42060)
0812 01:03:28.164432 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:28.164449 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:28.164685 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:28.165657 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:28.165661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:28.165690 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:28.166314 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:28.166326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.168371 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.168375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:28.170639 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:28.170652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.172424 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.172430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:28.172603 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:28.172645 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 01:03:28.173850 (+ 1205us) client_negotiation.cc:240] Negotiation successful
0812 01:03:28.173999 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":2,"thread_start_us":377,"threads_started":1}
I20260812 01:03:28.177407 20300 tablet_service.cc:1467] Processing CreateTablet for tablet 3eab1c89cdd44ecab683c54c1c4841d1 (DEFAULT_TABLE table=test-table [id=6fef0599c85146528f57e0c3e4db93c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:03:28.177876 20741 tablet_service.cc:1467] Processing CreateTablet for tablet 3eab1c89cdd44ecab683c54c1c4841d1 (DEFAULT_TABLE table=test-table [id=6fef0599c85146528f57e0c3e4db93c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:03:28.178180 20528 tablet_service.cc:1467] Processing CreateTablet for tablet 3eab1c89cdd44ecab683c54c1c4841d1 (DEFAULT_TABLE table=test-table [id=6fef0599c85146528f57e0c3e4db93c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:03:28.179109 20300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eab1c89cdd44ecab683c54c1c4841d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:03:28.179306 20741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eab1c89cdd44ecab683c54c1c4841d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:03:28.179491 20528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eab1c89cdd44ecab683c54c1c4841d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:03:28.192370 20881 tablet_bootstrap.cc:492] T 3eab1c89cdd44ecab683c54c1c4841d1 P ceb49de3cb994c55a33a2c5bc86a591b: Bootstrap starting.
I20260812 01:03:28.194152 20882 tablet_bootstrap.cc:492] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595: Bootstrap starting.
I20260812 01:03:28.195565 20884 tablet_bootstrap.cc:492] T 3eab1c89cdd44ecab683c54c1c4841d1 P 7de1baf8631f475d8a984ba7afffe1b6: Bootstrap starting.
I20260812 01:03:28.197484 20881 tablet_bootstrap.cc:654] T 3eab1c89cdd44ecab683c54c1c4841d1 P ceb49de3cb994c55a33a2c5bc86a591b: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:28.199311 20884 tablet_bootstrap.cc:654] T 3eab1c89cdd44ecab683c54c1c4841d1 P 7de1baf8631f475d8a984ba7afffe1b6: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:28.200536 20884 log.cc:824] T 3eab1c89cdd44ecab683c54c1c4841d1 P 7de1baf8631f475d8a984ba7afffe1b6: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:28.201479 20881 log.cc:824] T 3eab1c89cdd44ecab683c54c1c4841d1 P ceb49de3cb994c55a33a2c5bc86a591b: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:28.201727 20882 tablet_bootstrap.cc:654] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:28.203151 20882 log.cc:824] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:28.207554 20884 tablet_bootstrap.cc:492] T 3eab1c89cdd44ecab683c54c1c4841d1 P 7de1baf8631f475d8a984ba7afffe1b6: No bootstrap required, opened a new log
I20260812 01:03:28.207937 20884 ts_tablet_manager.cc:1397] T 3eab1c89cdd44ecab683c54c1c4841d1 P 7de1baf8631f475d8a984ba7afffe1b6: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20260812 01:03:28.208537 20882 tablet_bootstrap.cc:492] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595: No bootstrap required, opened a new log
I20260812 01:03:28.208861 20882 ts_tablet_manager.cc:1397] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.008s
I20260812 01:03:28.217625 20881 tablet_bootstrap.cc:492] T 3eab1c89cdd44ecab683c54c1c4841d1 P ceb49de3cb994c55a33a2c5bc86a591b: No bootstrap required, opened a new log
I20260812 01:03:28.217938 20881 ts_tablet_manager.cc:1397] T 3eab1c89cdd44ecab683c54c1c4841d1 P ceb49de3cb994c55a33a2c5bc86a591b: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.009s
I20260812 01:03:28.220588 20884 raft_consensus.cc:348] T 3eab1c89cdd44ecab683c54c1c4841d1 P 7de1baf8631f475d8a984ba7afffe1b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de1baf8631f475d8a984ba7afffe1b6" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33495 } } peers { permanent_uuid: "6bc61720a6064cb390db4072579ab595" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 39589 } } peers { permanent_uuid: "ceb49de3cb994c55a33a2c5bc86a591b" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34405 } }
I20260812 01:03:28.221091 20884 raft_consensus.cc:374] T 3eab1c89cdd44ecab683c54c1c4841d1 P 7de1baf8631f475d8a984ba7afffe1b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:28.221197 20884 raft_consensus.cc:729] T 3eab1c89cdd44ecab683c54c1c4841d1 P 7de1baf8631f475d8a984ba7afffe1b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7de1baf8631f475d8a984ba7afffe1b6, State: Initialized, Role: FOLLOWER
I20260812 01:03:28.221880 20884 consensus_queue.cc:260] T 3eab1c89cdd44ecab683c54c1c4841d1 P 7de1baf8631f475d8a984ba7afffe1b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de1baf8631f475d8a984ba7afffe1b6" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33495 } } peers { permanent_uuid: "6bc61720a6064cb390db4072579ab595" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 39589 } } peers { permanent_uuid: "ceb49de3cb994c55a33a2c5bc86a591b" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34405 } }
I20260812 01:03:28.224435 20882 raft_consensus.cc:348] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de1baf8631f475d8a984ba7afffe1b6" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33495 } } peers { permanent_uuid: "6bc61720a6064cb390db4072579ab595" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 39589 } } peers { permanent_uuid: "ceb49de3cb994c55a33a2c5bc86a591b" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34405 } }
I20260812 01:03:28.225099 20882 raft_consensus.cc:374] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:28.225273 20882 raft_consensus.cc:729] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bc61720a6064cb390db4072579ab595, State: Initialized, Role: FOLLOWER
I20260812 01:03:28.226207 20882 consensus_queue.cc:260] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de1baf8631f475d8a984ba7afffe1b6" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33495 } } peers { permanent_uuid: "6bc61720a6064cb390db4072579ab595" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 39589 } } peers { permanent_uuid: "ceb49de3cb994c55a33a2c5bc86a591b" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34405 } }
I20260812 01:03:28.231541 20881 raft_consensus.cc:348] T 3eab1c89cdd44ecab683c54c1c4841d1 P ceb49de3cb994c55a33a2c5bc86a591b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de1baf8631f475d8a984ba7afffe1b6" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33495 } } peers { permanent_uuid: "6bc61720a6064cb390db4072579ab595" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 39589 } } peers { permanent_uuid: "ceb49de3cb994c55a33a2c5bc86a591b" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34405 } }
I20260812 01:03:28.232019 20881 raft_consensus.cc:374] T 3eab1c89cdd44ecab683c54c1c4841d1 P ceb49de3cb994c55a33a2c5bc86a591b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:28.232129 20881 raft_consensus.cc:729] T 3eab1c89cdd44ecab683c54c1c4841d1 P ceb49de3cb994c55a33a2c5bc86a591b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb49de3cb994c55a33a2c5bc86a591b, State: Initialized, Role: FOLLOWER
I20260812 01:03:28.232800 20882 ts_tablet_manager.cc:1428] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595: Time spent starting tablet: real 0.024s user 0.018s sys 0.001s
I20260812 01:03:28.232843 20881 consensus_queue.cc:260] T 3eab1c89cdd44ecab683c54c1c4841d1 P ceb49de3cb994c55a33a2c5bc86a591b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de1baf8631f475d8a984ba7afffe1b6" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33495 } } peers { permanent_uuid: "6bc61720a6064cb390db4072579ab595" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 39589 } } peers { permanent_uuid: "ceb49de3cb994c55a33a2c5bc86a591b" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34405 } }
I20260812 01:03:28.234037 20850 heartbeater.cc:503] Master 127.21.201.126:38499 was elected leader, sending a full tablet report...
I20260812 01:03:28.234670 20884 ts_tablet_manager.cc:1428] T 3eab1c89cdd44ecab683c54c1c4841d1 P 7de1baf8631f475d8a984ba7afffe1b6: Time spent starting tablet: real 0.027s user 0.013s sys 0.013s
I20260812 01:03:28.235051 20631 heartbeater.cc:503] Master 127.21.201.126:38499 was elected leader, sending a full tablet report...
I20260812 01:03:28.236382 20881 ts_tablet_manager.cc:1428] T 3eab1c89cdd44ecab683c54c1c4841d1 P ceb49de3cb994c55a33a2c5bc86a591b: Time spent starting tablet: real 0.018s user 0.009s sys 0.006s
W20260812 01:03:28.250002 20633 tablet.cc:2369] T 3eab1c89cdd44ecab683c54c1c4841d1 P 7de1baf8631f475d8a984ba7afffe1b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:03:28.326516 20853 tablet.cc:2369] T 3eab1c89cdd44ecab683c54c1c4841d1 P ceb49de3cb994c55a33a2c5bc86a591b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:03:28.454857 20407 tablet.cc:2369] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:03:28.477571 20892 raft_consensus.cc:482] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:03:28.477919 20892 raft_consensus.cc:504] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de1baf8631f475d8a984ba7afffe1b6" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33495 } } peers { permanent_uuid: "6bc61720a6064cb390db4072579ab595" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 39589 } } peers { permanent_uuid: "ceb49de3cb994c55a33a2c5bc86a591b" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34405 } }
I20260812 01:03:28.479779 20892 leader_election.cc:288] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7de1baf8631f475d8a984ba7afffe1b6 (127.21.201.66:33495), ceb49de3cb994c55a33a2c5bc86a591b (127.21.201.67:34405)
I20260812 01:03:28.487416 20878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:28.480469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:56695 (local address 127.21.201.67:34405)
0812 01:03:28.480828 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:28.480851 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:28.481719 (+ 868us) server_negotiation.cc:407] Connection header received
0812 01:03:28.481964 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:28.481970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:28.482046 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:28.482149 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:28.482842 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.483613 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.485603 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.486400 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.486593 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:28.486678 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:28.487004 (+ 326us) server_negotiation.cc:299] Negotiation successful
0812 01:03:28.487139 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260812 01:03:28.487504 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:28.480361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:56927 (local address 127.21.201.66:33495)
0812 01:03:28.480652 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:28.480661 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:28.481729 (+ 1068us) server_negotiation.cc:407] Connection header received
0812 01:03:28.482007 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:28.482017 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:28.482081 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:28.482184 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:28.482900 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.483621 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.485608 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.486488 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.486668 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:28.486751 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:28.487056 (+ 305us) server_negotiation.cc:299] Negotiation successful
0812 01:03:28.487188 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260812 01:03:28.487591 20894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:28.480369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:34405 (local address 127.21.201.65:56695)
0812 01:03:28.481572 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:28.481589 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:28.481789 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:28.482175 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:28.482179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:28.482203 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:28.482683 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:28.482694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.483786 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.483792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:28.485446 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:28.485458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.486636 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.486642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:28.486829 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:28.486854 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 01:03:28.486967 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 01:03:28.487106 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1039,"mutex_wait_us":46,"thread_start_us":501,"threads_started":1}
I20260812 01:03:28.487982 20895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:28.480438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:33495 (local address 127.21.201.65:56927)
0812 01:03:28.481614 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:28.481630 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:28.481847 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:28.482199 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:28.482203 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:28.482226 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:28.482695 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:28.482706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.483801 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.483810 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:28.485444 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:28.485457 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.486726 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.486732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:28.486892 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:28.486915 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 01:03:28.487016 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 01:03:28.487144 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"mutex_wait_us":9,"spinlock_wait_cycles":24320,"thread_start_us":713,"threads_started":1}
I20260812 01:03:28.488859 20775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eab1c89cdd44ecab683c54c1c4841d1" candidate_uuid: "6bc61720a6064cb390db4072579ab595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb49de3cb994c55a33a2c5bc86a591b" is_pre_election: true
I20260812 01:03:28.489374 20775 raft_consensus.cc:2454] T 3eab1c89cdd44ecab683c54c1c4841d1 P ceb49de3cb994c55a33a2c5bc86a591b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bc61720a6064cb390db4072579ab595 in term 0.
I20260812 01:03:28.489249 20558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eab1c89cdd44ecab683c54c1c4841d1" candidate_uuid: "6bc61720a6064cb390db4072579ab595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7de1baf8631f475d8a984ba7afffe1b6" is_pre_election: true
I20260812 01:03:28.489785 20558 raft_consensus.cc:2454] T 3eab1c89cdd44ecab683c54c1c4841d1 P 7de1baf8631f475d8a984ba7afffe1b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bc61720a6064cb390db4072579ab595 in term 0.
I20260812 01:03:28.490085 20217 leader_election.cc:302] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bc61720a6064cb390db4072579ab595, ceb49de3cb994c55a33a2c5bc86a591b; no voters:
I20260812 01:03:28.490585 20892 raft_consensus.cc:2781] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:03:28.490746 20892 raft_consensus.cc:482] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:03:28.490878 20892 raft_consensus.cc:3037] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:28.493194 20892 raft_consensus.cc:504] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de1baf8631f475d8a984ba7afffe1b6" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33495 } } peers { permanent_uuid: "6bc61720a6064cb390db4072579ab595" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 39589 } } peers { permanent_uuid: "ceb49de3cb994c55a33a2c5bc86a591b" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34405 } }
I20260812 01:03:28.493901 20892 leader_election.cc:288] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595 [CANDIDATE]: Term 1 election: Requested vote from peers 7de1baf8631f475d8a984ba7afffe1b6 (127.21.201.66:33495), ceb49de3cb994c55a33a2c5bc86a591b (127.21.201.67:34405)
I20260812 01:03:28.494407 20775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eab1c89cdd44ecab683c54c1c4841d1" candidate_uuid: "6bc61720a6064cb390db4072579ab595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb49de3cb994c55a33a2c5bc86a591b"
I20260812 01:03:28.494444 20558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eab1c89cdd44ecab683c54c1c4841d1" candidate_uuid: "6bc61720a6064cb390db4072579ab595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7de1baf8631f475d8a984ba7afffe1b6"
I20260812 01:03:28.494628 20775 raft_consensus.cc:3037] T 3eab1c89cdd44ecab683c54c1c4841d1 P ceb49de3cb994c55a33a2c5bc86a591b [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:28.494673 20558 raft_consensus.cc:3037] T 3eab1c89cdd44ecab683c54c1c4841d1 P 7de1baf8631f475d8a984ba7afffe1b6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:28.497253 20775 raft_consensus.cc:2454] T 3eab1c89cdd44ecab683c54c1c4841d1 P ceb49de3cb994c55a33a2c5bc86a591b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bc61720a6064cb390db4072579ab595 in term 1.
I20260812 01:03:28.497650 20217 leader_election.cc:302] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bc61720a6064cb390db4072579ab595, ceb49de3cb994c55a33a2c5bc86a591b; no voters:
I20260812 01:03:28.497972 20892 raft_consensus.cc:2781] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:03:28.498263 20558 raft_consensus.cc:2454] T 3eab1c89cdd44ecab683c54c1c4841d1 P 7de1baf8631f475d8a984ba7afffe1b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bc61720a6064cb390db4072579ab595 in term 1.
I20260812 01:03:28.498656 20892 raft_consensus.cc:686] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595 [term 1 LEADER]: Becoming Leader. State: Replica: 6bc61720a6064cb390db4072579ab595, State: Running, Role: LEADER
I20260812 01:03:28.499218 20892 consensus_queue.cc:237] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de1baf8631f475d8a984ba7afffe1b6" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33495 } } peers { permanent_uuid: "6bc61720a6064cb390db4072579ab595" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 39589 } } peers { permanent_uuid: "ceb49de3cb994c55a33a2c5bc86a591b" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34405 } }
I20260812 01:03:28.504539 20096 catalog_manager.cc:5495] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bc61720a6064cb390db4072579ab595 (127.21.201.65). New cstate: current_term: 1 leader_uuid: "6bc61720a6064cb390db4072579ab595" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de1baf8631f475d8a984ba7afffe1b6" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bc61720a6064cb390db4072579ab595" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 39589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb49de3cb994c55a33a2c5bc86a591b" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34405 } health_report { overall_health: UNKNOWN } } }
I20260812 01:03:28.547919 20875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:28.542013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54720 (local address 127.21.201.65:39589)
0812 01:03:28.542252 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:28.542261 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:28.542299 (+ 38us) server_negotiation.cc:407] Connection header received
0812 01:03:28.542563 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:28.542567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:28.542650 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:28.542811 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:03:28.543304 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.544149 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.545680 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.546210 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.546308 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:28.546844 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:03:28.547349 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:03:28.547458 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:28.547571 (+ 113us) server_negotiation.cc:299] Negotiation successful
0812 01:03:28.547641 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 01:03:28.556803 20558 raft_consensus.cc:1261] T 3eab1c89cdd44ecab683c54c1c4841d1 P 7de1baf8631f475d8a984ba7afffe1b6 [term 1 FOLLOWER]: Refusing update from remote peer 6bc61720a6064cb390db4072579ab595: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:03:28.556792 20775 raft_consensus.cc:1261] T 3eab1c89cdd44ecab683c54c1c4841d1 P ceb49de3cb994c55a33a2c5bc86a591b [term 1 FOLLOWER]: Refusing update from remote peer 6bc61720a6064cb390db4072579ab595: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:03:28.557940 20896 consensus_queue.cc:1035] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7de1baf8631f475d8a984ba7afffe1b6" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 33495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:03:28.558363 20892 consensus_queue.cc:1035] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb49de3cb994c55a33a2c5bc86a591b" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 34405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:03:28.570406 20901 mvcc.cc:205] Tried to move back new op lower bound from 7317490108636979200 to 7317490108417384448. Current Snapshot: MvccSnapshot[applied={T|T < 7317490108636979200}]
I20260812 01:03:28.585216 20096 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:03:28.585505 20096 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:03:28.589506 20096 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e26c18bf67e4bef80964d1cba273c16: Sending DeleteTablet for 3 replicas of tablet 3eab1c89cdd44ecab683c54c1c4841d1
I20260812 01:03:28.590834 20528 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eab1c89cdd44ecab683c54c1c4841d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:03:28 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:38066
I20260812 01:03:28.590972 20300 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eab1c89cdd44ecab683c54c1c4841d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:03:28 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:54708
I20260812 01:03:28.590960 20741 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eab1c89cdd44ecab683c54c1c4841d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:03:28 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:42060
I20260812 01:03:28.593801 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 20188
I20260812 01:03:28.604049 20923 tablet_replica.cc:330] T 3eab1c89cdd44ecab683c54c1c4841d1 P ceb49de3cb994c55a33a2c5bc86a591b: stopping tablet replica
I20260812 01:03:28.604359 20922 tablet_replica.cc:330] T 3eab1c89cdd44ecab683c54c1c4841d1 P 7de1baf8631f475d8a984ba7afffe1b6: stopping tablet replica
I20260812 01:03:28.604827 20923 raft_consensus.cc:2229] T 3eab1c89cdd44ecab683c54c1c4841d1 P ceb49de3cb994c55a33a2c5bc86a591b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:03:28.604827 20922 raft_consensus.cc:2229] T 3eab1c89cdd44ecab683c54c1c4841d1 P 7de1baf8631f475d8a984ba7afffe1b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:03:28.605429 20923 raft_consensus.cc:2258] T 3eab1c89cdd44ecab683c54c1c4841d1 P ceb49de3cb994c55a33a2c5bc86a591b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:03:28.605429 20922 raft_consensus.cc:2258] T 3eab1c89cdd44ecab683c54c1c4841d1 P 7de1baf8631f475d8a984ba7afffe1b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:03:28.608349 20922 ts_tablet_manager.cc:1905] T 3eab1c89cdd44ecab683c54c1c4841d1 P 7de1baf8631f475d8a984ba7afffe1b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:03:28.608299 20923 ts_tablet_manager.cc:1905] T 3eab1c89cdd44ecab683c54c1c4841d1 P ceb49de3cb994c55a33a2c5bc86a591b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:03:28.609316 20924 tablet_replica.cc:330] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595: stopping tablet replica
I20260812 01:03:28.609810 20924 raft_consensus.cc:2229] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595 [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:03:28.610678 20924 raft_consensus.cc:2258] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:03:28.615794 20922 ts_tablet_manager.cc:1918] T 3eab1c89cdd44ecab683c54c1c4841d1 P 7de1baf8631f475d8a984ba7afffe1b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:03:28.615988 20922 log.cc:1196] T 3eab1c89cdd44ecab683c54c1c4841d1 P 7de1baf8631f475d8a984ba7afffe1b6: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/3eab1c89cdd44ecab683c54c1c4841d1
I20260812 01:03:28.616611 20922 ts_tablet_manager.cc:1939] T 3eab1c89cdd44ecab683c54c1c4841d1 P 7de1baf8631f475d8a984ba7afffe1b6: Deleting consensus metadata
I20260812 01:03:28.617647 20924 ts_tablet_manager.cc:1905] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:03:28.618790 20073 catalog_manager.cc:4841] TS 7de1baf8631f475d8a984ba7afffe1b6 (127.21.201.66:33495): tablet 3eab1c89cdd44ecab683c54c1c4841d1 (table test-table [id=6fef0599c85146528f57e0c3e4db93c7]) successfully deleted
I20260812 01:03:28.619999 20923 ts_tablet_manager.cc:1918] T 3eab1c89cdd44ecab683c54c1c4841d1 P ceb49de3cb994c55a33a2c5bc86a591b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:03:28.620177 20923 log.cc:1196] T 3eab1c89cdd44ecab683c54c1c4841d1 P ceb49de3cb994c55a33a2c5bc86a591b: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/3eab1c89cdd44ecab683c54c1c4841d1
I20260812 01:03:28.620780 20923 ts_tablet_manager.cc:1939] T 3eab1c89cdd44ecab683c54c1c4841d1 P ceb49de3cb994c55a33a2c5bc86a591b: Deleting consensus metadata
Aug 12 01:03:28 dist-test-slave-7f01 krb5kdc[20022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.65@KRBTEST.COM
I20260812 01:03:28.622686 20072 catalog_manager.cc:4841] TS ceb49de3cb994c55a33a2c5bc86a591b (127.21.201.67:34405): tablet 3eab1c89cdd44ecab683c54c1c4841d1 (table test-table [id=6fef0599c85146528f57e0c3e4db93c7]) successfully deleted
I20260812 01:03:28.625924 20924 ts_tablet_manager.cc:1918] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:03:28.626166 20924 log.cc:1196] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786496278741276-22309-0/minicluster-data/ts-0/wal/wals/3eab1c89cdd44ecab683c54c1c4841d1
I20260812 01:03:28.626655 20924 ts_tablet_manager.cc:1939] T 3eab1c89cdd44ecab683c54c1c4841d1 P 6bc61720a6064cb390db4072579ab595: Deleting consensus metadata
I20260812 01:03:28.628300 20072 catalog_manager.cc:4841] TS 6bc61720a6064cb390db4072579ab595 (127.21.201.65:39589): tablet 3eab1c89cdd44ecab683c54c1c4841d1 (table test-table [id=6fef0599c85146528f57e0c3e4db93c7]) successfully deleted
I20260812 01:03:28.631487 20875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:28.595664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54722 (local address 127.21.201.65:39589)
0812 01:03:28.595923 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:28.595933 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:28.608349 (+ 12416us) server_negotiation.cc:407] Connection header received
0812 01:03:28.608593 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:28.608600 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:28.608713 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:28.608833 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:28.611292 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.612492 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.614546 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.615405 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.615542 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:28.623313 (+ 7771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:28.623333 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:28.623352 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:28.623411 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:28.628083 (+ 4672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:28.628942 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:28.628951 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:28.628956 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:28.629206 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:28.629655 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:28.629660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:28.629664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:28.630146 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:28.630450 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:28.630790 (+ 340us) server_negotiation.cc:299] Negotiation successful
0812 01:03:28.631077 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 01:03:28.632402 20401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:28.736049 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 20188
I20260812 01:03:28.761459 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 20418
Aug 12 01:03:28 dist-test-slave-7f01 krb5kdc[20022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.66@KRBTEST.COM
I20260812 01:03:28.780387 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:28.762340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38078 (local address 127.21.201.66:33495)
0812 01:03:28.762558 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:28.762564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:28.762587 (+ 23us) server_negotiation.cc:407] Connection header received
0812 01:03:28.762778 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:28.762783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:28.762850 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:28.763003 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:28.764407 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.765222 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.766533 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.766951 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.767093 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:28.774944 (+ 7851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:28.774962 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:28.774979 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:28.775027 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:28.777879 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:28.778547 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:28.778555 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:28.778557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:28.778688 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:28.779020 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:28.779023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:28.779025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:28.779390 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:28.779637 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:28.779934 (+ 297us) server_negotiation.cc:299] Negotiation successful
0812 01:03:28.780107 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 01:03:28.781111 20625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:28.876873 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 20418
I20260812 01:03:28.905678 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 20638
Aug 12 01:03:28 dist-test-slave-7f01 krb5kdc[20022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.67@KRBTEST.COM
I20260812 01:03:28.921681 20878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:28.906575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42064 (local address 127.21.201.67:34405)
0812 01:03:28.906774 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:28.906779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:28.906799 (+ 20us) server_negotiation.cc:407] Connection header received
0812 01:03:28.906962 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:28.906967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:28.907045 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:28.907136 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:28.908623 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.909344 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.910656 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:28.911050 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:28.911146 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:28.916084 (+ 4938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:28.916109 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:28.916125 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:28.916169 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:28.918839 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:28.919561 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:28.919569 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:28.919571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:28.919727 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:28.920281 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:28.920284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:28.920286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:28.920694 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:28.920934 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:28.921223 (+ 289us) server_negotiation.cc:299] Negotiation successful
0812 01:03:28.921403 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 01:03:28.922540 20841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:29.015558 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 20638
I20260812 01:03:29.040398 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 20049
I20260812 01:03:29.041642 20146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:29.159202 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 20049
2026-08-12T01:03:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (5056 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-12T01:03:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:03:29Z Disabled control of system clock
I20260812 01:03:29.263594 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:42995
--webserver_interface=127.21.201.126
--webserver_port=0
--builtin_ntp_servers=127.21.201.84:43957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:42995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 01:03:29.613440 20942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:29.613749 20942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:29.613837 20942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:29.621829 20942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:03:29.621949 20942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:29.621987 20942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:29.622053 20942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:03:29.622102 20942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:03:29.633041 20942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:43957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:42995
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:42995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:29.634397 20942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:29.636329 20942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:29.648653 20954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:29.648504 20957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:29.648751 20955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:29.649811 20942 server_base.cc:1034] running on GCE node
I20260812 01:03:29.650652 20942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:29.652321 20942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:29.653561 20942 hybrid_clock.cc:648] HybridClock initialized: now 1786496609653526 us; error 57 us; skew 500 ppm
I20260812 01:03:29.653947 20942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:03:29.656383 20942 webserver.cc:460] Webserver started at http://127.21.201.126:36225/ using document root <none> and password file <none>
I20260812 01:03:29.657085 20942 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:29.657223 20942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:29.657573 20942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:29.660640 20942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "76217fd220574998b10315e7512e0b28"
format_stamp: "Formatted at 2026-08-12 01:03:29 on dist-test-slave-7f01"
server_key: "3764ec23720f0bde11ac0ba23a5ec38a"
server_key_iv: "3020d62e0dd1ba9c9ea16b1d9c352558"
server_key_version: "encryptionkey@0"
I20260812 01:03:29.661499 20942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "76217fd220574998b10315e7512e0b28"
format_stamp: "Formatted at 2026-08-12 01:03:29 on dist-test-slave-7f01"
server_key: "3764ec23720f0bde11ac0ba23a5ec38a"
server_key_iv: "3020d62e0dd1ba9c9ea16b1d9c352558"
server_key_version: "encryptionkey@0"
I20260812 01:03:29.666663 20942 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 01:03:29.670511 20967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:29.672017 20942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:03:29.672315 20942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "76217fd220574998b10315e7512e0b28"
format_stamp: "Formatted at 2026-08-12 01:03:29 on dist-test-slave-7f01"
server_key: "3764ec23720f0bde11ac0ba23a5ec38a"
server_key_iv: "3020d62e0dd1ba9c9ea16b1d9c352558"
server_key_version: "encryptionkey@0"
I20260812 01:03:29.672560 20942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:29.705086 20942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:29.706071 20942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:29.706394 20942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:29.726601 20942 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:42995
I20260812 01:03:29.726632 21054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:42995 every 8 connection(s)
I20260812 01:03:29.728407 20942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:03:29.732718 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 20942
I20260812 01:03:29.732914 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:03:29.733139 21055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:03:29.733446 22309 external_mini_cluster.cc:1428] Setting key 1d4ec609582521f43b8621881074e9a0
I20260812 01:03:29.749159 21055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76217fd220574998b10315e7512e0b28: Bootstrap starting.
I20260812 01:03:29.749426 21059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:29.735570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59140 (local address 127.21.201.126:42995)
0812 01:03:29.736774 (+ 1204us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:29.736789 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:29.736846 (+ 57us) server_negotiation.cc:407] Connection header received
0812 01:03:29.738796 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:29.738829 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:29.739444 (+ 615us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:29.739916 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:29.740494 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:29.742483 (+ 1989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:29.744083 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:29.745158 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:29.745354 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:29.746521 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:29.746534 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:29.746552 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:29.747316 (+ 764us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:29.747410 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:29.747435 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:29.747563 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:29.747810 (+ 247us) server_negotiation.cc:299] Negotiation successful
0812 01:03:29.748128 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":434,"threads_started":1}
I20260812 01:03:29.752444 21055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76217fd220574998b10315e7512e0b28: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:29.753372 21055 log.cc:824] T 00000000000000000000000000000000 P 76217fd220574998b10315e7512e0b28: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:29.756194 21055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76217fd220574998b10315e7512e0b28: No bootstrap required, opened a new log
I20260812 01:03:29.763334 21055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76217fd220574998b10315e7512e0b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76217fd220574998b10315e7512e0b28" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42995 } }
I20260812 01:03:29.763693 21055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76217fd220574998b10315e7512e0b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:29.763818 21055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76217fd220574998b10315e7512e0b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76217fd220574998b10315e7512e0b28, State: Initialized, Role: FOLLOWER
I20260812 01:03:29.764374 21055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76217fd220574998b10315e7512e0b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76217fd220574998b10315e7512e0b28" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42995 } }
I20260812 01:03:29.764613 21055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76217fd220574998b10315e7512e0b28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:03:29.764771 21055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76217fd220574998b10315e7512e0b28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:03:29.764940 21055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76217fd220574998b10315e7512e0b28 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:29.767136 21055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76217fd220574998b10315e7512e0b28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76217fd220574998b10315e7512e0b28" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42995 } }
I20260812 01:03:29.767644 21055 leader_election.cc:302] T 00000000000000000000000000000000 P 76217fd220574998b10315e7512e0b28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76217fd220574998b10315e7512e0b28; no voters:
I20260812 01:03:29.768332 21055 leader_election.cc:288] T 00000000000000000000000000000000 P 76217fd220574998b10315e7512e0b28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:03:29.768507 21061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76217fd220574998b10315e7512e0b28 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:03:29.769299 21061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76217fd220574998b10315e7512e0b28 [term 1 LEADER]: Becoming Leader. State: Replica: 76217fd220574998b10315e7512e0b28, State: Running, Role: LEADER
I20260812 01:03:29.769793 21061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76217fd220574998b10315e7512e0b28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76217fd220574998b10315e7512e0b28" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42995 } }
I20260812 01:03:29.770430 21055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76217fd220574998b10315e7512e0b28 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:03:29.772691 21064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76217fd220574998b10315e7512e0b28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76217fd220574998b10315e7512e0b28. Latest consensus state: current_term: 1 leader_uuid: "76217fd220574998b10315e7512e0b28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76217fd220574998b10315e7512e0b28" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42995 } } }
I20260812 01:03:29.772823 21062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76217fd220574998b10315e7512e0b28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76217fd220574998b10315e7512e0b28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76217fd220574998b10315e7512e0b28" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42995 } } }
I20260812 01:03:29.773167 21062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76217fd220574998b10315e7512e0b28 [sys.catalog]: This master's current role is: LEADER
I20260812 01:03:29.773164 21064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76217fd220574998b10315e7512e0b28 [sys.catalog]: This master's current role is: LEADER
I20260812 01:03:29.774261 21068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:03:29.780938 21068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:03:29.793210 21068 catalog_manager.cc:1347] Generated new cluster ID: 1fd3487fcbc24de99e14e793c21bfcba
I20260812 01:03:29.793354 21068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:03:29.810803 21068 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:03:29.812305 21068 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:03:29.822510 21068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76217fd220574998b10315e7512e0b28: Generated new TSK 0
I20260812 01:03:29.823380 21068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:03:29.828450 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:0
--local_ip_for_outbound_sockets=127.21.201.65
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:42995
--builtin_ntp_servers=127.21.201.84:43957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 01:03:30.203987 21087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:30.204269 21087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:30.204363 21087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:30.211697 21087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:30.211872 21087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:30.211999 21087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 01:03:30.221823 21087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:43957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:42995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:30.223083 21087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:30.224967 21087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:30.239223 21098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:30.239187 21102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:30.239099 21099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:30.240077 21087 server_base.cc:1034] running on GCE node
I20260812 01:03:30.240792 21087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:30.241868 21087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:30.243095 21087 hybrid_clock.cc:648] HybridClock initialized: now 1786496610243031 us; error 53 us; skew 500 ppm
I20260812 01:03:30.243434 21087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:03:30.245966 21087 webserver.cc:460] Webserver started at http://127.21.201.65:35749/ using document root <none> and password file <none>
I20260812 01:03:30.246802 21087 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:30.246937 21087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:30.247370 21087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:30.250995 21087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-0/data/instance:
uuid: "4b13c2ae1d28420e9776d0b4b792ac9e"
format_stamp: "Formatted at 2026-08-12 01:03:30 on dist-test-slave-7f01"
server_key: "1e15a5fe70588e15c5b1bfe747674b48"
server_key_iv: "3efaca642aef0f5d107c83708636d422"
server_key_version: "encryptionkey@0"
I20260812 01:03:30.252173 21087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance:
uuid: "4b13c2ae1d28420e9776d0b4b792ac9e"
format_stamp: "Formatted at 2026-08-12 01:03:30 on dist-test-slave-7f01"
server_key: "1e15a5fe70588e15c5b1bfe747674b48"
server_key_iv: "3efaca642aef0f5d107c83708636d422"
server_key_version: "encryptionkey@0"
I20260812 01:03:30.258055 21087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260812 01:03:30.261958 21113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:30.263326 21087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 01:03:30.263538 21087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "4b13c2ae1d28420e9776d0b4b792ac9e"
format_stamp: "Formatted at 2026-08-12 01:03:30 on dist-test-slave-7f01"
server_key: "1e15a5fe70588e15c5b1bfe747674b48"
server_key_iv: "3efaca642aef0f5d107c83708636d422"
server_key_version: "encryptionkey@0"
I20260812 01:03:30.263844 21087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:30.286592 21087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:30.287539 21087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:30.287936 21087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:30.289093 21087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:03:30.290776 21087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:03:30.290912 21087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:30.291273 21087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:03:30.291414 21087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:30.326952 21087 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:43895
I20260812 01:03:30.327060 21303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:43895 every 8 connection(s)
I20260812 01:03:30.328400 21087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 01:03:30.335902 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 21087
I20260812 01:03:30.336102 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance
I20260812 01:03:30.336562 22309 external_mini_cluster.cc:1428] Setting key 343f8fd45a72a43fef9b95cd6d4d6162
I20260812 01:03:30.339615 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:0
--local_ip_for_outbound_sockets=127.21.201.66
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:42995
--builtin_ntp_servers=127.21.201.84:43957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 01:03:30.341925 21308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:30.331220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:58737 (local address 127.21.201.126:42995)
0812 01:03:30.331726 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:30.331731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:30.332756 (+ 1025us) server_negotiation.cc:407] Connection header received
0812 01:03:30.334610 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:30.334616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:30.334720 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:30.334842 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:30.335940 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:30.336819 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:30.338956 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:30.339382 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:30.339514 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:30.341026 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:30.341032 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:30.341037 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:30.341061 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:30.341103 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:30.341115 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:30.341211 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:30.341447 (+ 236us) server_negotiation.cc:299] Negotiation successful
0812 01:03:30.341544 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":302,"threads_started":1}
I20260812 01:03:30.342849 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:30.331642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:42995 (local address 127.21.201.65:58737)
0812 01:03:30.332595 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:30.332640 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:30.334341 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:30.335071 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:30.335078 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:30.335160 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:03:30.335708 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:30.335725 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:30.336971 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:30.336983 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:30.338807 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:30.338819 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:30.339607 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:30.339614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:30.339957 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:30.340685 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:03:30.340687 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:03:30.340725 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:03:30.340803 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:03:30.341210 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:03:30.341225 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:03:30.341235 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 01:03:30.341412 (+ 177us) client_negotiation.cc:240] Negotiation successful
0812 01:03:30.341790 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":285,"threads_started":1}
I20260812 01:03:30.344412 21306 heartbeater.cc:347] Connected to a master server at 127.21.201.126:42995
I20260812 01:03:30.344719 21306 heartbeater.cc:464] Registering TS with master...
I20260812 01:03:30.345445 21306 heartbeater.cc:511] Master 127.21.201.126:42995 requested a full tablet report, sending...
I20260812 01:03:30.347821 21000 ts_manager.cc:194] Registered new tserver with Master: 4b13c2ae1d28420e9776d0b4b792ac9e (127.21.201.65:43895)
I20260812 01:03:30.349977 21000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.201.65:58737
W20260812 01:03:30.689587 21311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:30.689918 21311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:30.690090 21311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:30.697383 21311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:30.697526 21311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:30.697703 21311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 01:03:30.707109 21311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:43957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:42995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:30.708501 21311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:30.710312 21311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:30.724793 21325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:30.724857 21331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:30.724761 21329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:30.725806 21311 server_base.cc:1034] running on GCE node
I20260812 01:03:30.726590 21311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:30.727900 21311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:30.729173 21311 hybrid_clock.cc:648] HybridClock initialized: now 1786496610729108 us; error 47 us; skew 500 ppm
I20260812 01:03:30.729573 21311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:03:30.732407 21311 webserver.cc:460] Webserver started at http://127.21.201.66:33797/ using document root <none> and password file <none>
I20260812 01:03:30.733265 21311 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:30.733381 21311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:30.733740 21311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:30.737519 21311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-1/data/instance:
uuid: "93e0d7d9f6e541feaf6c76ab40f5cdb0"
format_stamp: "Formatted at 2026-08-12 01:03:30 on dist-test-slave-7f01"
server_key: "46e9f28125e9da91f09a25f78b65db78"
server_key_iv: "743831b8fe594045698b4156397d8b06"
server_key_version: "encryptionkey@0"
I20260812 01:03:30.738480 21311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance:
uuid: "93e0d7d9f6e541feaf6c76ab40f5cdb0"
format_stamp: "Formatted at 2026-08-12 01:03:30 on dist-test-slave-7f01"
server_key: "46e9f28125e9da91f09a25f78b65db78"
server_key_iv: "743831b8fe594045698b4156397d8b06"
server_key_version: "encryptionkey@0"
I20260812 01:03:30.745438 21311 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260812 01:03:30.749842 21344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:30.751199 21311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 01:03:30.751400 21311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "93e0d7d9f6e541feaf6c76ab40f5cdb0"
format_stamp: "Formatted at 2026-08-12 01:03:30 on dist-test-slave-7f01"
server_key: "46e9f28125e9da91f09a25f78b65db78"
server_key_iv: "743831b8fe594045698b4156397d8b06"
server_key_version: "encryptionkey@0"
I20260812 01:03:30.751642 21311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:30.789443 21311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:30.790376 21311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:30.790717 21311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:30.791849 21311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:03:30.793407 21311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:03:30.793541 21311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:30.793684 21311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:03:30.793778 21311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:30.831583 21311 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:34543
I20260812 01:03:30.831681 21534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:34543 every 8 connection(s)
I20260812 01:03:30.833063 21311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 01:03:30.834507 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 21311
I20260812 01:03:30.834712 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance
I20260812 01:03:30.835309 22309 external_mini_cluster.cc:1428] Setting key 6cc3d8ab0fc3f0bbdab00fdda14ff152
I20260812 01:03:30.838874 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:0
--local_ip_for_outbound_sockets=127.21.201.67
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:42995
--builtin_ntp_servers=127.21.201.84:43957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 01:03:30.847126 21308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:30.836302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:37271 (local address 127.21.201.126:42995)
0812 01:03:30.836513 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:30.836518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:30.838191 (+ 1673us) server_negotiation.cc:407] Connection header received
0812 01:03:30.840220 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:30.840226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:30.840324 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:30.840442 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:30.841535 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:30.842277 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:30.844138 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:30.844590 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:30.844725 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:30.846242 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:30.846247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:30.846250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:30.846269 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:30.846314 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:30.846326 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:30.846430 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:30.846656 (+ 226us) server_negotiation.cc:299] Negotiation successful
0812 01:03:30.846776 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 01:03:30.848277 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:30.836903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:42995 (local address 127.21.201.66:37271)
0812 01:03:30.838036 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:30.838073 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:30.839946 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:30.840686 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:30.840695 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:30.840780 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:03:30.841362 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:30.841381 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:30.842443 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:30.842450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:30.843993 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:30.844006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:30.844834 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:30.844841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:30.845078 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:30.845862 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:03:30.845866 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:03:30.845916 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:03:30.846010 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:03:30.846429 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:03:30.846442 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:03:30.846452 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 01:03:30.846644 (+ 192us) client_negotiation.cc:240] Negotiation successful
0812 01:03:30.847098 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":372,"threads_started":1}
I20260812 01:03:30.849917 21535 heartbeater.cc:347] Connected to a master server at 127.21.201.126:42995
I20260812 01:03:30.850274 21535 heartbeater.cc:464] Registering TS with master...
I20260812 01:03:30.851075 21535 heartbeater.cc:511] Master 127.21.201.126:42995 requested a full tablet report, sending...
I20260812 01:03:30.852741 21000 ts_manager.cc:194] Registered new tserver with Master: 93e0d7d9f6e541feaf6c76ab40f5cdb0 (127.21.201.66:34543)
I20260812 01:03:30.854190 21000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.201.66:37271
W20260812 01:03:31.191365 21539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:31.191664 21539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:31.191833 21539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:31.200732 21539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:31.200834 21539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:31.200955 21539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 01:03:31.210240 21539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:43957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:42995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:31.211521 21539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:31.213423 21539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:31.226575 21546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:31.226642 21551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:31.227174 21547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:31.228469 21539 server_base.cc:1034] running on GCE node
I20260812 01:03:31.229157 21539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:31.230167 21539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:31.231424 21539 hybrid_clock.cc:648] HybridClock initialized: now 1786496611231358 us; error 43 us; skew 500 ppm
I20260812 01:03:31.231909 21539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:03:31.234401 21539 webserver.cc:460] Webserver started at http://127.21.201.67:40745/ using document root <none> and password file <none>
I20260812 01:03:31.235150 21539 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:31.235286 21539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:31.235630 21539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:31.238595 21539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-2/data/instance:
uuid: "432acd04f92a44f08b9c7fe5b9a5b970"
format_stamp: "Formatted at 2026-08-12 01:03:31 on dist-test-slave-7f01"
server_key: "cca106919b5cea6af7961b05cabaa44b"
server_key_iv: "9733f5a312a6a8147218dc9b7edf04f2"
server_key_version: "encryptionkey@0"
I20260812 01:03:31.239436 21539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance:
uuid: "432acd04f92a44f08b9c7fe5b9a5b970"
format_stamp: "Formatted at 2026-08-12 01:03:31 on dist-test-slave-7f01"
server_key: "cca106919b5cea6af7961b05cabaa44b"
server_key_iv: "9733f5a312a6a8147218dc9b7edf04f2"
server_key_version: "encryptionkey@0"
I20260812 01:03:31.245222 21539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260812 01:03:31.249158 21563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:31.250528 21539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:03:31.250731 21539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "432acd04f92a44f08b9c7fe5b9a5b970"
format_stamp: "Formatted at 2026-08-12 01:03:31 on dist-test-slave-7f01"
server_key: "cca106919b5cea6af7961b05cabaa44b"
server_key_iv: "9733f5a312a6a8147218dc9b7edf04f2"
server_key_version: "encryptionkey@0"
I20260812 01:03:31.250995 21539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:31.280839 21539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:31.281833 21539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:31.282199 21539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:31.283435 21539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:03:31.285161 21539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:03:31.285316 21539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:31.285471 21539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:03:31.285573 21539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:31.322724 21539 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:37273
I20260812 01:03:31.322789 21750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:37273 every 8 connection(s)
I20260812 01:03:31.324262 21539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 01:03:31.333690 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 21539
I20260812 01:03:31.333899 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance
I20260812 01:03:31.334407 22309 external_mini_cluster.cc:1428] Setting key e68b2cbbb176c040ddbc312fe0908e61
I20260812 01:03:31.337738 21308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:31.327036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:33399 (local address 127.21.201.126:42995)
0812 01:03:31.327273 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:31.327280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:31.328856 (+ 1576us) server_negotiation.cc:407] Connection header received
0812 01:03:31.330668 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:31.330675 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:31.330773 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:31.330890 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:31.332097 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.332882 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.334668 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.335115 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.335281 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:31.336820 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:31.336825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:31.336829 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:31.336847 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:31.336881 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:31.336893 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:31.337012 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:31.337269 (+ 257us) server_negotiation.cc:299] Negotiation successful
0812 01:03:31.337373 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 01:03:31.338723 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:31.327541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:42995 (local address 127.21.201.67:33399)
0812 01:03:31.328645 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:31.328699 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:31.330363 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:31.331129 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:31.331137 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:31.331217 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:03:31.331913 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:31.331932 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.333077 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.333081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:31.334489 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:31.334530 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.335358 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.335368 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:31.335696 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:31.336500 (+ 804us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:03:31.336502 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:03:31.336543 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:03:31.336616 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:03:31.337038 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:03:31.337049 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:03:31.337064 (+ 15us) client_negotiation.cc:769] Sending connection context
0812 01:03:31.337235 (+ 171us) client_negotiation.cc:240] Negotiation successful
0812 01:03:31.337626 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":416,"threads_started":1}
I20260812 01:03:31.340221 21751 heartbeater.cc:347] Connected to a master server at 127.21.201.126:42995
I20260812 01:03:31.340529 21751 heartbeater.cc:464] Registering TS with master...
I20260812 01:03:31.341306 21751 heartbeater.cc:511] Master 127.21.201.126:42995 requested a full tablet report, sending...
I20260812 01:03:31.342762 21000 ts_manager.cc:194] Registered new tserver with Master: 432acd04f92a44f08b9c7fe5b9a5b970 (127.21.201.67:37273)
I20260812 01:03:31.343611 21000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.201.67:33399
I20260812 01:03:31.349905 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:03:31.353695 21306 heartbeater.cc:503] Master 127.21.201.126:42995 was elected leader, sending a full tablet report...
I20260812 01:03:31.360808 21308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:31.354281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59152 (local address 127.21.201.126:42995)
0812 01:03:31.354523 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:31.354531 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:31.354962 (+ 431us) server_negotiation.cc:407] Connection header received
0812 01:03:31.355172 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:31.355177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:31.355265 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:31.355357 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:31.356000 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.356943 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.358363 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.358784 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.358884 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:31.359980 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:31.359985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:31.359989 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:31.360008 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:31.360046 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:31.360060 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:31.360145 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:31.360324 (+ 179us) server_negotiation.cc:299] Negotiation successful
0812 01:03:31.360409 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 01:03:31.365175 21000 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:03:31.372623 21000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:03:31.400777 21770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:31.385409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:34543 (local address 127.0.0.1:35252)
0812 01:03:31.386183 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:31.386236 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:31.386404 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:31.393077 (+ 6673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:31.393086 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:31.393135 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:31.393745 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:31.393764 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.395679 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.395684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:31.397857 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:31.397869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.399574 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.399580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:31.399859 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:31.399932 (+ 73us) client_negotiation.cc:769] Sending connection context
0812 01:03:31.400081 (+ 149us) client_negotiation.cc:240] Negotiation successful
0812 01:03:31.400305 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":58,"thread_start_us":402,"threads_started":1}
I20260812 01:03:31.402014 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:31.385573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35252 (local address 127.21.201.66:34543)
0812 01:03:31.392478 (+ 6905us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:31.392486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:31.392514 (+ 28us) server_negotiation.cc:407] Connection header received
0812 01:03:31.392611 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:31.392617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:31.392847 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:31.393038 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:31.393944 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.395522 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.398076 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.399332 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.401097 (+ 1765us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:31.401345 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:31.401402 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 01:03:31.401617 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6739,"thread_start_us":6604,"threads_started":1}
I20260812 01:03:31.402164 21779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:31.385422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:43895 (local address 127.0.0.1:41610)
0812 01:03:31.387112 (+ 1690us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:31.387135 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:31.387254 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:31.394330 (+ 7076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:31.394335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:31.394363 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:31.394744 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:31.394755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.397059 (+ 2304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.397066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:31.399237 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:31.399251 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.401191 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.401201 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:31.401432 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:31.401471 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 01:03:31.401548 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 01:03:31.401712 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1545,"mutex_wait_us":234,"thread_start_us":6201,"threads_started":1}
I20260812 01:03:31.402189 21772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:31.386572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:37273 (local address 127.0.0.1:44548)
0812 01:03:31.391644 (+ 5072us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:31.391664 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:31.391922 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:31.392988 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:31.392997 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:31.393028 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:31.393555 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:31.393576 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.395552 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.395559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:31.398587 (+ 3028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:31.398602 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.401240 (+ 2638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.401245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:31.401492 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:31.401526 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 01:03:31.401624 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 01:03:31.401759 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4945,"mutex_wait_us":157,"thread_start_us":1672,"threads_started":1}
I20260812 01:03:31.402861 21780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:31.387246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44548 (local address 127.21.201.67:37273)
0812 01:03:31.388061 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:31.388070 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:31.391806 (+ 3736us) server_negotiation.cc:407] Connection header received
0812 01:03:31.392147 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:31.392158 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:31.392431 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:31.392602 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:31.393815 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.395399 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.399022 (+ 3623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.400524 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.400665 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:31.401519 (+ 854us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:31.401681 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 01:03:31.401847 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":459,"threads_started":1}
I20260812 01:03:31.403358 21778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:31.385823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41610 (local address 127.21.201.65:43895)
0812 01:03:31.393717 (+ 7894us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:31.393728 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:31.393758 (+ 30us) server_negotiation.cc:407] Connection header received
0812 01:03:31.393879 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:31.393886 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:31.394114 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:31.394339 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:31.395174 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.396863 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.399462 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.400968 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.402437 (+ 1469us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:31.402722 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:31.402810 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 01:03:31.402995 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7692,"thread_start_us":7524,"threads_started":1}
I20260812 01:03:31.405853 21419 tablet_service.cc:1467] Processing CreateTablet for tablet 2db7af8bcb0447e6862cb15ad3300dab (DEFAULT_TABLE table=test-table [id=d2de754acc1545f89a5cfa7b6c1ba62f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:03:31.406422 21643 tablet_service.cc:1467] Processing CreateTablet for tablet 2db7af8bcb0447e6862cb15ad3300dab (DEFAULT_TABLE table=test-table [id=d2de754acc1545f89a5cfa7b6c1ba62f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:03:31.406731 21190 tablet_service.cc:1467] Processing CreateTablet for tablet 2db7af8bcb0447e6862cb15ad3300dab (DEFAULT_TABLE table=test-table [id=d2de754acc1545f89a5cfa7b6c1ba62f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:03:31.407233 21419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db7af8bcb0447e6862cb15ad3300dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:03:31.407946 21643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db7af8bcb0447e6862cb15ad3300dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:03:31.408044 21190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db7af8bcb0447e6862cb15ad3300dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:03:31.418812 21781 tablet_bootstrap.cc:492] T 2db7af8bcb0447e6862cb15ad3300dab P 93e0d7d9f6e541feaf6c76ab40f5cdb0: Bootstrap starting.
I20260812 01:03:31.422313 21788 tablet_bootstrap.cc:492] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e: Bootstrap starting.
I20260812 01:03:31.422808 21781 tablet_bootstrap.cc:654] T 2db7af8bcb0447e6862cb15ad3300dab P 93e0d7d9f6e541feaf6c76ab40f5cdb0: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:31.424422 21781 log.cc:824] T 2db7af8bcb0447e6862cb15ad3300dab P 93e0d7d9f6e541feaf6c76ab40f5cdb0: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:31.426292 21788 tablet_bootstrap.cc:654] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:31.427194 21789 tablet_bootstrap.cc:492] T 2db7af8bcb0447e6862cb15ad3300dab P 432acd04f92a44f08b9c7fe5b9a5b970: Bootstrap starting.
I20260812 01:03:31.427632 21788 log.cc:824] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:31.428400 21781 tablet_bootstrap.cc:492] T 2db7af8bcb0447e6862cb15ad3300dab P 93e0d7d9f6e541feaf6c76ab40f5cdb0: No bootstrap required, opened a new log
I20260812 01:03:31.428709 21781 ts_tablet_manager.cc:1397] T 2db7af8bcb0447e6862cb15ad3300dab P 93e0d7d9f6e541feaf6c76ab40f5cdb0: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260812 01:03:31.430850 21788 tablet_bootstrap.cc:492] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e: No bootstrap required, opened a new log
I20260812 01:03:31.430886 21789 tablet_bootstrap.cc:654] T 2db7af8bcb0447e6862cb15ad3300dab P 432acd04f92a44f08b9c7fe5b9a5b970: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:31.431146 21788 ts_tablet_manager.cc:1397] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260812 01:03:31.432163 21789 log.cc:824] T 2db7af8bcb0447e6862cb15ad3300dab P 432acd04f92a44f08b9c7fe5b9a5b970: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:31.435456 21789 tablet_bootstrap.cc:492] T 2db7af8bcb0447e6862cb15ad3300dab P 432acd04f92a44f08b9c7fe5b9a5b970: No bootstrap required, opened a new log
I20260812 01:03:31.435784 21789 ts_tablet_manager.cc:1397] T 2db7af8bcb0447e6862cb15ad3300dab P 432acd04f92a44f08b9c7fe5b9a5b970: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260812 01:03:31.437649 21781 raft_consensus.cc:348] T 2db7af8bcb0447e6862cb15ad3300dab P 93e0d7d9f6e541feaf6c76ab40f5cdb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e0d7d9f6e541feaf6c76ab40f5cdb0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34543 } } peers { permanent_uuid: "4b13c2ae1d28420e9776d0b4b792ac9e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43895 } } peers { permanent_uuid: "432acd04f92a44f08b9c7fe5b9a5b970" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37273 } }
I20260812 01:03:31.438035 21781 raft_consensus.cc:374] T 2db7af8bcb0447e6862cb15ad3300dab P 93e0d7d9f6e541feaf6c76ab40f5cdb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:31.438165 21781 raft_consensus.cc:729] T 2db7af8bcb0447e6862cb15ad3300dab P 93e0d7d9f6e541feaf6c76ab40f5cdb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e0d7d9f6e541feaf6c76ab40f5cdb0, State: Initialized, Role: FOLLOWER
I20260812 01:03:31.438810 21781 consensus_queue.cc:260] T 2db7af8bcb0447e6862cb15ad3300dab P 93e0d7d9f6e541feaf6c76ab40f5cdb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e0d7d9f6e541feaf6c76ab40f5cdb0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34543 } } peers { permanent_uuid: "4b13c2ae1d28420e9776d0b4b792ac9e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43895 } } peers { permanent_uuid: "432acd04f92a44f08b9c7fe5b9a5b970" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37273 } }
I20260812 01:03:31.441792 21788 raft_consensus.cc:348] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e0d7d9f6e541feaf6c76ab40f5cdb0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34543 } } peers { permanent_uuid: "4b13c2ae1d28420e9776d0b4b792ac9e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43895 } } peers { permanent_uuid: "432acd04f92a44f08b9c7fe5b9a5b970" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37273 } }
I20260812 01:03:31.442248 21788 raft_consensus.cc:374] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:31.442348 21788 raft_consensus.cc:729] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b13c2ae1d28420e9776d0b4b792ac9e, State: Initialized, Role: FOLLOWER
I20260812 01:03:31.443167 21535 heartbeater.cc:503] Master 127.21.201.126:42995 was elected leader, sending a full tablet report...
I20260812 01:03:31.443073 21788 consensus_queue.cc:260] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e0d7d9f6e541feaf6c76ab40f5cdb0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34543 } } peers { permanent_uuid: "4b13c2ae1d28420e9776d0b4b792ac9e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43895 } } peers { permanent_uuid: "432acd04f92a44f08b9c7fe5b9a5b970" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37273 } }
I20260812 01:03:31.444343 21781 ts_tablet_manager.cc:1428] T 2db7af8bcb0447e6862cb15ad3300dab P 93e0d7d9f6e541feaf6c76ab40f5cdb0: Time spent starting tablet: real 0.015s user 0.009s sys 0.006s
I20260812 01:03:31.447130 21789 raft_consensus.cc:348] T 2db7af8bcb0447e6862cb15ad3300dab P 432acd04f92a44f08b9c7fe5b9a5b970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e0d7d9f6e541feaf6c76ab40f5cdb0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34543 } } peers { permanent_uuid: "4b13c2ae1d28420e9776d0b4b792ac9e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43895 } } peers { permanent_uuid: "432acd04f92a44f08b9c7fe5b9a5b970" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37273 } }
I20260812 01:03:31.447542 21788 ts_tablet_manager.cc:1428] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
I20260812 01:03:31.447685 21789 raft_consensus.cc:374] T 2db7af8bcb0447e6862cb15ad3300dab P 432acd04f92a44f08b9c7fe5b9a5b970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:31.447834 21789 raft_consensus.cc:729] T 2db7af8bcb0447e6862cb15ad3300dab P 432acd04f92a44f08b9c7fe5b9a5b970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 432acd04f92a44f08b9c7fe5b9a5b970, State: Initialized, Role: FOLLOWER
I20260812 01:03:31.448431 21789 consensus_queue.cc:260] T 2db7af8bcb0447e6862cb15ad3300dab P 432acd04f92a44f08b9c7fe5b9a5b970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e0d7d9f6e541feaf6c76ab40f5cdb0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34543 } } peers { permanent_uuid: "4b13c2ae1d28420e9776d0b4b792ac9e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43895 } } peers { permanent_uuid: "432acd04f92a44f08b9c7fe5b9a5b970" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37273 } }
I20260812 01:03:31.450312 21751 heartbeater.cc:503] Master 127.21.201.126:42995 was elected leader, sending a full tablet report...
I20260812 01:03:31.450856 21789 ts_tablet_manager.cc:1428] T 2db7af8bcb0447e6862cb15ad3300dab P 432acd04f92a44f08b9c7fe5b9a5b970: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
W20260812 01:03:31.576339 21752 tablet.cc:2369] T 2db7af8bcb0447e6862cb15ad3300dab P 432acd04f92a44f08b9c7fe5b9a5b970: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:03:31.581032 21307 tablet.cc:2369] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:03:31.585695 21536 tablet.cc:2369] T 2db7af8bcb0447e6862cb15ad3300dab P 93e0d7d9f6e541feaf6c76ab40f5cdb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:03:31.710206 21800 raft_consensus.cc:482] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:03:31.710551 21800 raft_consensus.cc:504] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e0d7d9f6e541feaf6c76ab40f5cdb0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34543 } } peers { permanent_uuid: "4b13c2ae1d28420e9776d0b4b792ac9e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43895 } } peers { permanent_uuid: "432acd04f92a44f08b9c7fe5b9a5b970" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37273 } }
I20260812 01:03:31.712308 21800 leader_election.cc:288] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93e0d7d9f6e541feaf6c76ab40f5cdb0 (127.21.201.66:34543), 432acd04f92a44f08b9c7fe5b9a5b970 (127.21.201.67:37273)
I20260812 01:03:31.720224 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:31.712842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:56901 (local address 127.21.201.66:34543)
0812 01:03:31.713111 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:31.713120 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:31.713704 (+ 584us) server_negotiation.cc:407] Connection header received
0812 01:03:31.713962 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:31.713968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:31.714082 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:31.714155 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:31.715110 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.715956 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.718045 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.719167 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.719296 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:31.719351 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:31.719800 (+ 449us) server_negotiation.cc:299] Negotiation successful
0812 01:03:31.719970 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 01:03:31.720323 21802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:31.712834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:34543 (local address 127.21.201.65:56901)
0812 01:03:31.713533 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:31.713550 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:31.713746 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:31.714393 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:31.714400 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:31.714498 (+ 98us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:31.714939 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:31.714953 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.716098 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.716104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:31.717839 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:31.717852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.719367 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.719375 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:31.719547 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:31.719589 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 01:03:31.719698 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 01:03:31.719866 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":65,"spinlock_wait_cycles":23296,"thread_start_us":410,"threads_started":1}
I20260812 01:03:31.720899 21780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:31.713976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:54547 (local address 127.21.201.67:37273)
0812 01:03:31.714190 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:31.714196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:31.714422 (+ 226us) server_negotiation.cc:407] Connection header received
0812 01:03:31.714544 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:31.714552 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:31.714638 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:31.714739 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:31.715360 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.716211 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.718637 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.719677 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.719865 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:31.719930 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:31.720475 (+ 545us) server_negotiation.cc:299] Negotiation successful
0812 01:03:31.720599 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 01:03:31.721421 21804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:31.713632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:37273 (local address 127.21.201.65:54547)
0812 01:03:31.714302 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:31.714324 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:31.714434 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:31.714763 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:31.714766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:31.714788 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:31.715190 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:31.715202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.716460 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.716465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:31.718423 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:31.718440 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.720060 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.720065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:31.720252 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:31.720303 (+ 51us) client_negotiation.cc:769] Sending connection context
0812 01:03:31.721037 (+ 734us) client_negotiation.cc:240] Negotiation successful
0812 01:03:31.721177 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":4,"thread_start_us":337,"threads_started":1}
I20260812 01:03:31.721586 21464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db7af8bcb0447e6862cb15ad3300dab" candidate_uuid: "4b13c2ae1d28420e9776d0b4b792ac9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e0d7d9f6e541feaf6c76ab40f5cdb0" is_pre_election: true
I20260812 01:03:31.722226 21464 raft_consensus.cc:2454] T 2db7af8bcb0447e6862cb15ad3300dab P 93e0d7d9f6e541feaf6c76ab40f5cdb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b13c2ae1d28420e9776d0b4b792ac9e in term 0.
I20260812 01:03:31.722322 21676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db7af8bcb0447e6862cb15ad3300dab" candidate_uuid: "4b13c2ae1d28420e9776d0b4b792ac9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "432acd04f92a44f08b9c7fe5b9a5b970" is_pre_election: true
I20260812 01:03:31.722867 21676 raft_consensus.cc:2454] T 2db7af8bcb0447e6862cb15ad3300dab P 432acd04f92a44f08b9c7fe5b9a5b970 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b13c2ae1d28420e9776d0b4b792ac9e in term 0.
I20260812 01:03:31.722976 21114 leader_election.cc:302] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b13c2ae1d28420e9776d0b4b792ac9e, 93e0d7d9f6e541feaf6c76ab40f5cdb0; no voters:
I20260812 01:03:31.723474 21800 raft_consensus.cc:2781] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:03:31.723624 21800 raft_consensus.cc:482] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:03:31.723776 21800 raft_consensus.cc:3037] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:31.726050 21800 raft_consensus.cc:504] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e0d7d9f6e541feaf6c76ab40f5cdb0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34543 } } peers { permanent_uuid: "4b13c2ae1d28420e9776d0b4b792ac9e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43895 } } peers { permanent_uuid: "432acd04f92a44f08b9c7fe5b9a5b970" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37273 } }
I20260812 01:03:31.726701 21800 leader_election.cc:288] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e [CANDIDATE]: Term 1 election: Requested vote from peers 93e0d7d9f6e541feaf6c76ab40f5cdb0 (127.21.201.66:34543), 432acd04f92a44f08b9c7fe5b9a5b970 (127.21.201.67:37273)
I20260812 01:03:31.727332 21464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db7af8bcb0447e6862cb15ad3300dab" candidate_uuid: "4b13c2ae1d28420e9776d0b4b792ac9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e0d7d9f6e541feaf6c76ab40f5cdb0"
I20260812 01:03:31.727589 21464 raft_consensus.cc:3037] T 2db7af8bcb0447e6862cb15ad3300dab P 93e0d7d9f6e541feaf6c76ab40f5cdb0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:31.727671 21676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db7af8bcb0447e6862cb15ad3300dab" candidate_uuid: "4b13c2ae1d28420e9776d0b4b792ac9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "432acd04f92a44f08b9c7fe5b9a5b970"
I20260812 01:03:31.727911 21676 raft_consensus.cc:3037] T 2db7af8bcb0447e6862cb15ad3300dab P 432acd04f92a44f08b9c7fe5b9a5b970 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:31.730106 21464 raft_consensus.cc:2454] T 2db7af8bcb0447e6862cb15ad3300dab P 93e0d7d9f6e541feaf6c76ab40f5cdb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b13c2ae1d28420e9776d0b4b792ac9e in term 1.
I20260812 01:03:31.730587 21114 leader_election.cc:302] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b13c2ae1d28420e9776d0b4b792ac9e, 93e0d7d9f6e541feaf6c76ab40f5cdb0; no voters:
I20260812 01:03:31.730895 21800 raft_consensus.cc:2781] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:03:31.731209 21676 raft_consensus.cc:2454] T 2db7af8bcb0447e6862cb15ad3300dab P 432acd04f92a44f08b9c7fe5b9a5b970 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b13c2ae1d28420e9776d0b4b792ac9e in term 1.
I20260812 01:03:31.731704 21800 raft_consensus.cc:686] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e [term 1 LEADER]: Becoming Leader. State: Replica: 4b13c2ae1d28420e9776d0b4b792ac9e, State: Running, Role: LEADER
I20260812 01:03:31.732627 21800 consensus_queue.cc:237] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e0d7d9f6e541feaf6c76ab40f5cdb0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34543 } } peers { permanent_uuid: "4b13c2ae1d28420e9776d0b4b792ac9e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43895 } } peers { permanent_uuid: "432acd04f92a44f08b9c7fe5b9a5b970" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37273 } }
I20260812 01:03:31.737365 20997 catalog_manager.cc:5495] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b13c2ae1d28420e9776d0b4b792ac9e (127.21.201.65). New cstate: current_term: 1 leader_uuid: "4b13c2ae1d28420e9776d0b4b792ac9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e0d7d9f6e541feaf6c76ab40f5cdb0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b13c2ae1d28420e9776d0b4b792ac9e" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "432acd04f92a44f08b9c7fe5b9a5b970" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37273 } health_report { overall_health: UNKNOWN } } }
I20260812 01:03:31.765555 21778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:31.760033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41626 (local address 127.21.201.65:43895)
0812 01:03:31.760233 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:31.760237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:31.760257 (+ 20us) server_negotiation.cc:407] Connection header received
0812 01:03:31.760398 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:31.760402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:31.760496 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:31.760639 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:03:31.761107 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.761869 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.763365 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.763891 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.764001 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:31.764523 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:03:31.765078 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:03:31.765223 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:31.765267 (+ 44us) server_negotiation.cc:299] Negotiation successful
0812 01:03:31.765349 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 01:03:31.774362 21676 raft_consensus.cc:1261] T 2db7af8bcb0447e6862cb15ad3300dab P 432acd04f92a44f08b9c7fe5b9a5b970 [term 1 FOLLOWER]: Refusing update from remote peer 4b13c2ae1d28420e9776d0b4b792ac9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:03:31.774384 21464 raft_consensus.cc:1261] T 2db7af8bcb0447e6862cb15ad3300dab P 93e0d7d9f6e541feaf6c76ab40f5cdb0 [term 1 FOLLOWER]: Refusing update from remote peer 4b13c2ae1d28420e9776d0b4b792ac9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:03:31.775377 21806 consensus_queue.cc:1035] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e0d7d9f6e541feaf6c76ab40f5cdb0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 34543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:03:31.775820 21800 consensus_queue.cc:1035] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "432acd04f92a44f08b9c7fe5b9a5b970" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:03:31.791069 21813 mvcc.cc:205] Tried to move back new op lower bound from 7317490121817649152 to 7317490121661001728. Current Snapshot: MvccSnapshot[applied={T|T < 7317490121817649152}]
I20260812 01:03:31.792887 21810 mvcc.cc:205] Tried to move back new op lower bound from 7317490121817649152 to 7317490121661001728. Current Snapshot: MvccSnapshot[applied={T|T < 7317490121817649152}]
I20260812 01:03:31.804575 20997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:03:31.804891 20997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:03:31.808823 20997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76217fd220574998b10315e7512e0b28: Sending DeleteTablet for 3 replicas of tablet 2db7af8bcb0447e6862cb15ad3300dab
I20260812 01:03:31.810137 21190 tablet_service.cc:1514] Processing DeleteTablet for tablet 2db7af8bcb0447e6862cb15ad3300dab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:03:31 UTC) from {username='slave'} at 127.0.0.1:41610
I20260812 01:03:31.810129 21643 tablet_service.cc:1514] Processing DeleteTablet for tablet 2db7af8bcb0447e6862cb15ad3300dab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:03:31 UTC) from {username='slave'} at 127.0.0.1:44548
I20260812 01:03:31.811056 21419 tablet_service.cc:1514] Processing DeleteTablet for tablet 2db7af8bcb0447e6862cb15ad3300dab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:03:31 UTC) from {username='slave'} at 127.0.0.1:35252
I20260812 01:03:31.812345 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 21087
I20260812 01:03:31.821574 21825 tablet_replica.cc:330] T 2db7af8bcb0447e6862cb15ad3300dab P 93e0d7d9f6e541feaf6c76ab40f5cdb0: stopping tablet replica
I20260812 01:03:31.823113 21825 raft_consensus.cc:2229] T 2db7af8bcb0447e6862cb15ad3300dab P 93e0d7d9f6e541feaf6c76ab40f5cdb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:03:31.824803 21822 tablet_replica.cc:330] T 2db7af8bcb0447e6862cb15ad3300dab P 432acd04f92a44f08b9c7fe5b9a5b970: stopping tablet replica
I20260812 01:03:31.825701 21825 raft_consensus.cc:2258] T 2db7af8bcb0447e6862cb15ad3300dab P 93e0d7d9f6e541feaf6c76ab40f5cdb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:03:31.826411 21822 raft_consensus.cc:2229] T 2db7af8bcb0447e6862cb15ad3300dab P 432acd04f92a44f08b9c7fe5b9a5b970 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:03:31.828532 21822 raft_consensus.cc:2258] T 2db7af8bcb0447e6862cb15ad3300dab P 432acd04f92a44f08b9c7fe5b9a5b970 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:03:31.829865 21825 ts_tablet_manager.cc:1905] T 2db7af8bcb0447e6862cb15ad3300dab P 93e0d7d9f6e541feaf6c76ab40f5cdb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:03:31.831135 21778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:31.813611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41636 (local address 127.21.201.65:43895)
0812 01:03:31.813854 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:31.813863 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:31.820583 (+ 6720us) server_negotiation.cc:407] Connection header received
0812 01:03:31.821475 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:31.821482 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:31.821618 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:31.821737 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:31.823074 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.824403 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.826393 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.827134 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.827234 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:31.829267 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:31.829279 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:31.829307 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:31.830127 (+ 820us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:31.830190 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:31.830215 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:31.830294 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:31.830587 (+ 293us) server_negotiation.cc:299] Negotiation successful
0812 01:03:31.830705 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 01:03:31.842396 21825 ts_tablet_manager.cc:1918] T 2db7af8bcb0447e6862cb15ad3300dab P 93e0d7d9f6e541feaf6c76ab40f5cdb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:03:31.842818 21825 log.cc:1196] T 2db7af8bcb0447e6862cb15ad3300dab P 93e0d7d9f6e541feaf6c76ab40f5cdb0: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/2db7af8bcb0447e6862cb15ad3300dab
I20260812 01:03:31.843778 21825 ts_tablet_manager.cc:1939] T 2db7af8bcb0447e6862cb15ad3300dab P 93e0d7d9f6e541feaf6c76ab40f5cdb0: Deleting consensus metadata
I20260812 01:03:31.844981 21295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:31.846243 20970 catalog_manager.cc:4841] TS 93e0d7d9f6e541feaf6c76ab40f5cdb0 (127.21.201.66:34543): tablet 2db7af8bcb0447e6862cb15ad3300dab (table test-table [id=d2de754acc1545f89a5cfa7b6c1ba62f]) successfully deleted
I20260812 01:03:31.848408 21822 ts_tablet_manager.cc:1905] T 2db7af8bcb0447e6862cb15ad3300dab P 432acd04f92a44f08b9c7fe5b9a5b970: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:03:31.854437 21822 ts_tablet_manager.cc:1918] T 2db7af8bcb0447e6862cb15ad3300dab P 432acd04f92a44f08b9c7fe5b9a5b970: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:03:31.854722 21822 log.cc:1196] T 2db7af8bcb0447e6862cb15ad3300dab P 432acd04f92a44f08b9c7fe5b9a5b970: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/2db7af8bcb0447e6862cb15ad3300dab
I20260812 01:03:31.855324 21822 ts_tablet_manager.cc:1939] T 2db7af8bcb0447e6862cb15ad3300dab P 432acd04f92a44f08b9c7fe5b9a5b970: Deleting consensus metadata
I20260812 01:03:31.857285 20970 catalog_manager.cc:4841] TS 432acd04f92a44f08b9c7fe5b9a5b970 (127.21.201.67:37273): tablet 2db7af8bcb0447e6862cb15ad3300dab (table test-table [id=d2de754acc1545f89a5cfa7b6c1ba62f]) successfully deleted
I20260812 01:03:31.960078 21823 tablet_replica.cc:330] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e: stopping tablet replica
I20260812 01:03:31.960621 21823 raft_consensus.cc:2229] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:03:31.961338 21823 raft_consensus.cc:2258] T 2db7af8bcb0447e6862cb15ad3300dab P 4b13c2ae1d28420e9776d0b4b792ac9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:03:31.961388 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 21087
I20260812 01:03:31.986002 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 21311
W20260812 01:03:31.985975 20976 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.201.65:43895 (error 108)
I20260812 01:03:31.988049 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:31.987229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:43895 (local address 127.0.0.1:41650)
0812 01:03:31.987690 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:31.987877 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:43895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":253,"threads_started":1}
W20260812 01:03:31.988272 20976 catalog_manager.cc:4584] TS 4b13c2ae1d28420e9776d0b4b792ac9e (127.21.201.65:43895): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2db7af8bcb0447e6862cb15ad3300dab: Network error: Client connection negotiation failed: client connection to 127.21.201.65:43895: connect: Connection refused (error 111)
I20260812 01:03:31.993503 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:31.986837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.21.201.66:34543)
0812 01:03:31.987028 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:31.987034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:31.987155 (+ 121us) server_negotiation.cc:407] Connection header received
0812 01:03:31.987381 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:31.987386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:31.987469 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:31.987584 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:31.988057 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.988944 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.990249 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:31.990696 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:31.990848 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:31.991990 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:31.992002 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:31.992026 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:31.992683 (+ 657us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:31.992732 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:31.992754 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:31.992890 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:31.993035 (+ 145us) server_negotiation.cc:299] Negotiation successful
0812 01:03:31.993150 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 01:03:31.994244 21529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:32.041904 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:32.041406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:43895 (local address 127.0.0.1:41666)
0812 01:03:32.041640 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:32.041734 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:43895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260812 01:03:32.042999 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:32.042608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:43895 (local address 127.0.0.1:41674)
0812 01:03:32.042780 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:32.042848 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:43895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260812 01:03:32.087551 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 21311
I20260812 01:03:32.103950 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:32.103353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:43895 (local address 127.0.0.1:41676)
0812 01:03:32.103590 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:32.103694 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:43895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260812 01:03:32.105221 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:32.104794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:43895 (local address 127.0.0.1:41686)
0812 01:03:32.104965 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:32.105057 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:43895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260812 01:03:32.113639 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 21539
I20260812 01:03:32.121079 21780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:32.114522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44564 (local address 127.21.201.67:37273)
0812 01:03:32.114756 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:32.114793 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:32.114820 (+ 27us) server_negotiation.cc:407] Connection header received
0812 01:03:32.114903 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:32.114907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:32.115007 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:32.115137 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:32.115609 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:32.116528 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:32.117863 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:32.118289 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:32.118444 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:32.119625 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:32.119634 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:32.119649 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:32.120296 (+ 647us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:32.120344 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:32.120365 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:32.120508 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:32.120643 (+ 135us) server_negotiation.cc:299] Negotiation successful
0812 01:03:32.120755 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260812 01:03:32.121785 21744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:32.185835 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:32.185279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:43895 (local address 127.0.0.1:41696)
0812 01:03:32.185542 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:32.185666 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:43895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260812 01:03:32.187101 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:32.186607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:43895 (local address 127.0.0.1:41712)
0812 01:03:32.186874 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:32.186950 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:43895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260812 01:03:32.216027 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 21539
I20260812 01:03:32.239425 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 20942
I20260812 01:03:32.240772 21043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:32.358589 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 20942
2026-08-12T01:03:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3191 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:03:32 dist-test-slave-7f01 krb5kdc[21842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:03:32 dist-test-slave-7f01 krb5kdc[21842](info): set up 2 sockets
Aug 12 01:03:32 dist-test-slave-7f01 krb5kdc[21842](info): commencing operation
krb5kdc: starting...
W20260812 01:03:33.766173 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.356s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:03:33 dist-test-slave-7f01 krb5kdc[21842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:03:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:03:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 01:03:34.147644 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:36659
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:36659
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:03:34.623824 21865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:34.624189 21865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:34.624361 21865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:34.631693 21865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:03:34.631866 21865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:34.631980 21865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:34.632090 21865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:03:34.632175 21865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:03:34.641976 21865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:36659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:36659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:34.643342 21865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:34.644785 21865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:34.657423 21874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:34.657558 21875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:34.657620 21877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:34.658847 21865 server_base.cc:1034] running on GCE node
I20260812 01:03:34.659839 21865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:34.661512 21865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:34.662752 21865 hybrid_clock.cc:648] HybridClock initialized: now 1786496614662707 us; error 38 us; skew 500 ppm
I20260812 01:03:34.663177 21865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:03:34 dist-test-slave-7f01 krb5kdc[21842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496614, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:03:34.670449 21865 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 01:03:34.672328 21865 webserver.cc:460] Webserver started at http://127.21.201.126:38795/ using document root <none> and password file <none>
I20260812 01:03:34.673117 21865 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:34.673282 21865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:34.673638 21865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:34.677417 21865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "0f21c73e37ae4f4d9be848b1261c2aca"
format_stamp: "Formatted at 2026-08-12 01:03:34 on dist-test-slave-7f01"
server_key: "e47733c89ff5d6a21a4ff0f88b1d7c00"
server_key_iv: "506acec5ac004625d9c12da01bb802a2"
server_key_version: "encryptionkey@0"
I20260812 01:03:34.678272 21865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "0f21c73e37ae4f4d9be848b1261c2aca"
format_stamp: "Formatted at 2026-08-12 01:03:34 on dist-test-slave-7f01"
server_key: "e47733c89ff5d6a21a4ff0f88b1d7c00"
server_key_iv: "506acec5ac004625d9c12da01bb802a2"
server_key_version: "encryptionkey@0"
I20260812 01:03:34.684176 21865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 01:03:34.687943 21896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:34.689579 21865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:03:34.689793 21865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "0f21c73e37ae4f4d9be848b1261c2aca"
format_stamp: "Formatted at 2026-08-12 01:03:34 on dist-test-slave-7f01"
server_key: "e47733c89ff5d6a21a4ff0f88b1d7c00"
server_key_iv: "506acec5ac004625d9c12da01bb802a2"
server_key_version: "encryptionkey@0"
I20260812 01:03:34.690052 21865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:34.725623 21865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:34.730290 21865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:34.730633 21865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:34.751329 21865 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:36659
I20260812 01:03:34.751353 21987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:36659 every 8 connection(s)
I20260812 01:03:34.752851 21865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:03:34.754419 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 21865
I20260812 01:03:34.754622 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:03:34.755224 22309 external_mini_cluster.cc:1428] Setting key ce5d19e2b5dffc883065dad2a137562a
I20260812 01:03:34.757825 21989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 01:03:34 dist-test-slave-7f01 krb5kdc[21842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:03:34.772392 21989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f21c73e37ae4f4d9be848b1261c2aca: Bootstrap starting.
I20260812 01:03:34.775928 21989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f21c73e37ae4f4d9be848b1261c2aca: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:34.777007 21989 log.cc:824] T 00000000000000000000000000000000 P 0f21c73e37ae4f4d9be848b1261c2aca: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:34.780155 21989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f21c73e37ae4f4d9be848b1261c2aca: No bootstrap required, opened a new log
I20260812 01:03:34.780314 21994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:34.757339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42328 (local address 127.21.201.126:36659)
0812 01:03:34.758247 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:34.758259 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:34.758300 (+ 41us) server_negotiation.cc:407] Connection header received
0812 01:03:34.760121 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:34.760143 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:34.760531 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:34.760860 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:34.762906 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:34.764435 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:34.766423 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:34.767225 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:34.767436 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:34.772746 (+ 5310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:34.772754 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:34.772768 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:34.772808 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:34.776118 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:34.776895 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:34.776902 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:34.776908 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:34.777078 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:34.777491 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:34.777496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:34.777499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:34.778042 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:34.778393 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:34.778823 (+ 430us) server_negotiation.cc:299] Negotiation successful
0812 01:03:34.779264 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":436,"threads_started":1}
I20260812 01:03:34.788296 21989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f21c73e37ae4f4d9be848b1261c2aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f21c73e37ae4f4d9be848b1261c2aca" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 36659 } }
I20260812 01:03:34.788786 21989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f21c73e37ae4f4d9be848b1261c2aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:34.788902 21989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f21c73e37ae4f4d9be848b1261c2aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f21c73e37ae4f4d9be848b1261c2aca, State: Initialized, Role: FOLLOWER
I20260812 01:03:34.789629 21989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f21c73e37ae4f4d9be848b1261c2aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f21c73e37ae4f4d9be848b1261c2aca" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 36659 } }
I20260812 01:03:34.789920 21989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f21c73e37ae4f4d9be848b1261c2aca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:03:34.790071 21989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f21c73e37ae4f4d9be848b1261c2aca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:03:34.790205 21989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f21c73e37ae4f4d9be848b1261c2aca [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:34.792522 21989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f21c73e37ae4f4d9be848b1261c2aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f21c73e37ae4f4d9be848b1261c2aca" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 36659 } }
I20260812 01:03:34.792984 21989 leader_election.cc:302] T 00000000000000000000000000000000 P 0f21c73e37ae4f4d9be848b1261c2aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f21c73e37ae4f4d9be848b1261c2aca; no voters:
I20260812 01:03:34.793681 21989 leader_election.cc:288] T 00000000000000000000000000000000 P 0f21c73e37ae4f4d9be848b1261c2aca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:03:34.793824 21998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f21c73e37ae4f4d9be848b1261c2aca [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:03:34.794687 21998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f21c73e37ae4f4d9be848b1261c2aca [term 1 LEADER]: Becoming Leader. State: Replica: 0f21c73e37ae4f4d9be848b1261c2aca, State: Running, Role: LEADER
I20260812 01:03:34.795291 21998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f21c73e37ae4f4d9be848b1261c2aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f21c73e37ae4f4d9be848b1261c2aca" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 36659 } }
I20260812 01:03:34.796255 21989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f21c73e37ae4f4d9be848b1261c2aca [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:03:34.798276 22003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f21c73e37ae4f4d9be848b1261c2aca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f21c73e37ae4f4d9be848b1261c2aca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f21c73e37ae4f4d9be848b1261c2aca" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 36659 } } }
I20260812 01:03:34.798638 22003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f21c73e37ae4f4d9be848b1261c2aca [sys.catalog]: This master's current role is: LEADER
I20260812 01:03:34.799976 22004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f21c73e37ae4f4d9be848b1261c2aca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f21c73e37ae4f4d9be848b1261c2aca. Latest consensus state: current_term: 1 leader_uuid: "0f21c73e37ae4f4d9be848b1261c2aca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f21c73e37ae4f4d9be848b1261c2aca" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 36659 } } }
I20260812 01:03:34.800412 22013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:03:34.800468 22004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f21c73e37ae4f4d9be848b1261c2aca [sys.catalog]: This master's current role is: LEADER
I20260812 01:03:34.806123 22013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:03:34.815456 22013 catalog_manager.cc:1347] Generated new cluster ID: e1bafbd57b18447cbc84e8f54a8c32b7
I20260812 01:03:34.815588 22013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:03:34.832823 22013 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:03:34.834244 22013 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:03:34.854554 22013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f21c73e37ae4f4d9be848b1261c2aca: Generated new TSK 0
I20260812 01:03:34.855535 22013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.201.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
I20260812 01:03:34.943341 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:0
--local_ip_for_outbound_sockets=127.21.201.65
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:36659
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:03:35.306409 22042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:35.306728 22042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:35.306850 22042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:35.314428 22042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:35.314543 22042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:35.314661 22042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 01:03:35.324594 22042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:36659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:35.325830 22042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:35.327162 22042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:35.340934 22047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:35.340896 22048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:35.340968 22050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:35.342087 22042 server_base.cc:1034] running on GCE node
I20260812 01:03:35.342730 22042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:35.343717 22042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:35.345005 22042 hybrid_clock.cc:648] HybridClock initialized: now 1786496615344960 us; error 52 us; skew 500 ppm
I20260812 01:03:35.345333 22042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:03:35 dist-test-slave-7f01 krb5kdc[21842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496615, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:03:35.348667 22042 init.cc:377] Logged in from keytab as kudu/127.21.201.65@KRBTEST.COM (short username kudu)
I20260812 01:03:35.350132 22042 webserver.cc:460] Webserver started at http://127.21.201.65:45993/ using document root <none> and password file <none>
I20260812 01:03:35.350797 22042 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:35.350907 22042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:35.351188 22042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:35.354233 22042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-0/data/instance:
uuid: "94abb9efef014df0a0df400785c5893b"
format_stamp: "Formatted at 2026-08-12 01:03:35 on dist-test-slave-7f01"
server_key: "6de63c7a7d61c1a13c5fab0a6ba992e6"
server_key_iv: "a104b2585fd6e6e945bdfd61d0bf6a49"
server_key_version: "encryptionkey@0"
I20260812 01:03:35.355116 22042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance:
uuid: "94abb9efef014df0a0df400785c5893b"
format_stamp: "Formatted at 2026-08-12 01:03:35 on dist-test-slave-7f01"
server_key: "6de63c7a7d61c1a13c5fab0a6ba992e6"
server_key_iv: "a104b2585fd6e6e945bdfd61d0bf6a49"
server_key_version: "encryptionkey@0"
I20260812 01:03:35.360610 22042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260812 01:03:35.364477 22059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:35.365993 22042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:03:35.366132 22042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "94abb9efef014df0a0df400785c5893b"
format_stamp: "Formatted at 2026-08-12 01:03:35 on dist-test-slave-7f01"
server_key: "6de63c7a7d61c1a13c5fab0a6ba992e6"
server_key_iv: "a104b2585fd6e6e945bdfd61d0bf6a49"
server_key_version: "encryptionkey@0"
I20260812 01:03:35.366312 22042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:35.396143 22042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:35.400652 22042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:35.400933 22042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:35.401993 22042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:03:35.403350 22042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:03:35.403438 22042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:35.403524 22042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:03:35.403581 22042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:35.439163 22042 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:35939
I20260812 01:03:35.439224 22247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:35939 every 8 connection(s)
I20260812 01:03:35.440564 22042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 01:03:35.446457 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 22042
I20260812 01:03:35.446686 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance
I20260812 01:03:35.447228 22309 external_mini_cluster.cc:1428] Setting key 47cc1650574beb8b167581204183b8cc
Aug 12 01:03:35 dist-test-slave-7f01 krb5kdc[21842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496615, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:03:35.463234 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:35.443397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:36469 (local address 127.21.201.126:36659)
0812 01:03:35.443933 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:35.443938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:35.444796 (+ 858us) server_negotiation.cc:407] Connection header received
0812 01:03:35.446833 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:35.446839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:35.446945 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:35.447083 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:35.448981 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:35.449879 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:35.451516 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:35.452026 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:35.452162 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:35.456148 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:35.456157 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:35.456165 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:35.456214 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:35.459972 (+ 3758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:35.460768 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:35.460776 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:35.460782 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:35.460943 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:35.461471 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:35.461479 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:35.461485 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:35.461893 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:35.462054 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:35.462769 (+ 715us) server_negotiation.cc:299] Negotiation successful
0812 01:03:35.462954 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":260,"threads_started":1}
I20260812 01:03:35.464809 22254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:35.443902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:36659 (local address 127.21.201.65:36469)
0812 01:03:35.444623 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:35.444668 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:35.446486 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:35.447389 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:35.447402 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:35.448192 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:03:35.448790 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:35.448809 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:35.450057 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:35.450061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:35.451347 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:35.451361 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:35.452285 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:35.452293 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:35.452538 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:35.453301 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:03:35.453325 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:03:35.455902 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:03:35.460157 (+ 4255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:03:35.460164 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:03:35.460176 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:03:35.460626 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:03:35.461113 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:03:35.461118 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:03:35.461120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:03:35.461334 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:03:35.462028 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:03:35.462034 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:03:35.462357 (+ 323us) client_negotiation.cc:769] Sending connection context
0812 01:03:35.462586 (+ 229us) client_negotiation.cc:240] Negotiation successful
0812 01:03:35.463243 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":299,"threads_started":1}
I20260812 01:03:35.466439 22248 heartbeater.cc:347] Connected to a master server at 127.21.201.126:36659
I20260812 01:03:35.466778 22248 heartbeater.cc:464] Registering TS with master...
I20260812 01:03:35.467537 22248 heartbeater.cc:511] Master 127.21.201.126:36659 requested a full tablet report, sending...
I20260812 01:03:35.469857 21938 ts_manager.cc:194] Registered new tserver with Master: 94abb9efef014df0a0df400785c5893b (127.21.201.65:35939)
I20260812 01:03:35.471385 21938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.65@KRBTEST.COM'} at 127.21.201.65:36469
WARNING: no policy specified for kudu/127.21.201.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
I20260812 01:03:35.521354 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:0
--local_ip_for_outbound_sockets=127.21.201.66
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:36659
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:03:35.899132 22258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:35.899505 22258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:35.899696 22258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:35.908252 22258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:35.908427 22258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:35.908624 22258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 01:03:35.919704 22258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:36659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:35.921321 22258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:35.922785 22258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:35.938151 22267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:35.938230 22271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:35.938227 22266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:35.939136 22258 server_base.cc:1034] running on GCE node
I20260812 01:03:35.939813 22258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:35.940768 22258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:35.941988 22258 hybrid_clock.cc:648] HybridClock initialized: now 1786496615941937 us; error 67 us; skew 500 ppm
I20260812 01:03:35.942277 22258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:03:35 dist-test-slave-7f01 krb5kdc[21842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496615, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:03:35.945722 22258 init.cc:377] Logged in from keytab as kudu/127.21.201.66@KRBTEST.COM (short username kudu)
I20260812 01:03:35.947309 22258 webserver.cc:460] Webserver started at http://127.21.201.66:46443/ using document root <none> and password file <none>
I20260812 01:03:35.948084 22258 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:35.948191 22258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:35.948448 22258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:35.951323 22258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-1/data/instance:
uuid: "332a52b29b1c4c329dc16a1877de4c81"
format_stamp: "Formatted at 2026-08-12 01:03:35 on dist-test-slave-7f01"
server_key: "cc88f16df8ba175b55eb264af7196cdb"
server_key_iv: "334d4b93acd7f2755d3688bb0dd3f773"
server_key_version: "encryptionkey@0"
I20260812 01:03:35.952133 22258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance:
uuid: "332a52b29b1c4c329dc16a1877de4c81"
format_stamp: "Formatted at 2026-08-12 01:03:35 on dist-test-slave-7f01"
server_key: "cc88f16df8ba175b55eb264af7196cdb"
server_key_iv: "334d4b93acd7f2755d3688bb0dd3f773"
server_key_version: "encryptionkey@0"
I20260812 01:03:35.957257 22258 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260812 01:03:35.960669 22284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:35.961992 22258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:03:35.962177 22258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "332a52b29b1c4c329dc16a1877de4c81"
format_stamp: "Formatted at 2026-08-12 01:03:35 on dist-test-slave-7f01"
server_key: "cc88f16df8ba175b55eb264af7196cdb"
server_key_iv: "334d4b93acd7f2755d3688bb0dd3f773"
server_key_version: "encryptionkey@0"
I20260812 01:03:35.962378 22258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:35.976701 22258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:35.982267 22258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:35.982645 22258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:35.984041 22258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:03:35.985952 22258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:03:35.986063 22258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:35.986212 22258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:03:35.986287 22258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:36.021777 22258 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:35871
I20260812 01:03:36.021816 22480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:35871 every 8 connection(s)
I20260812 01:03:36.023164 22258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 01:03:36.032701 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 22258
I20260812 01:03:36.032923 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance
I20260812 01:03:36.033425 22309 external_mini_cluster.cc:1428] Setting key e6a2db47d2903d717fc10c60dd3346f1
Aug 12 01:03:36 dist-test-slave-7f01 krb5kdc[21842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496615, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:03:36.044844 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:36.026003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:44631 (local address 127.21.201.126:36659)
0812 01:03:36.026544 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:36.026550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:36.027400 (+ 850us) server_negotiation.cc:407] Connection header received
0812 01:03:36.029390 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:36.029396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:36.029506 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:36.029633 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:36.031207 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.032070 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.033806 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.034353 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.034491 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:36.038640 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:36.038645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:36.038649 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:36.038681 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:36.041534 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:36.042319 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:36.042325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:36.042329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:36.042469 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:36.042929 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:36.042935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:36.042938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:36.043240 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:36.043404 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:36.044207 (+ 803us) server_negotiation.cc:299] Negotiation successful
0812 01:03:36.044449 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":314,"threads_started":1}
I20260812 01:03:36.045879 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:36.026414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:36659 (local address 127.21.201.66:44631)
0812 01:03:36.027219 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:36.027261 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:36.029106 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:36.029875 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:36.029883 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:36.030444 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:03:36.031026 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:36.031045 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.032230 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.032235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:36.033663 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:36.033677 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.034615 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.034624 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:36.034918 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:36.035800 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:03:36.035834 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:03:36.038384 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:03:36.041710 (+ 3326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:03:36.041719 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:03:36.041736 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:03:36.042187 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:03:36.042596 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:03:36.042601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:03:36.042605 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:03:36.042808 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:03:36.043376 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:03:36.043384 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:03:36.043821 (+ 437us) client_negotiation.cc:769] Sending connection context
0812 01:03:36.044109 (+ 288us) client_negotiation.cc:240] Negotiation successful
0812 01:03:36.044690 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":324,"threads_started":1}
I20260812 01:03:36.047453 22481 heartbeater.cc:347] Connected to a master server at 127.21.201.126:36659
I20260812 01:03:36.047842 22481 heartbeater.cc:464] Registering TS with master...
I20260812 01:03:36.048609 22481 heartbeater.cc:511] Master 127.21.201.126:36659 requested a full tablet report, sending...
I20260812 01:03:36.050143 21938 ts_manager.cc:194] Registered new tserver with Master: 332a52b29b1c4c329dc16a1877de4c81 (127.21.201.66:35871)
I20260812 01:03:36.050988 21938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.66@KRBTEST.COM'} at 127.21.201.66:44631
WARNING: no policy specified for kudu/127.21.201.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
I20260812 01:03:36.109380 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:0
--local_ip_for_outbound_sockets=127.21.201.67
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:36659
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:39413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:03:36.452646 22493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:36.452919 22493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:36.453075 22493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:36.460518 22493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:36.460657 22493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:36.460845 22493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 01:03:36.471091 22493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:39413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:36659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:36.472612 22493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:36.473968 22493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 01:03:36.474455 22248 heartbeater.cc:503] Master 127.21.201.126:36659 was elected leader, sending a full tablet report...
W20260812 01:03:36.487902 22507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:36.487949 22504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:36.487914 22511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:36.489295 22493 server_base.cc:1034] running on GCE node
I20260812 01:03:36.489928 22493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:36.490864 22493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:36.492180 22493 hybrid_clock.cc:648] HybridClock initialized: now 1786496616492101 us; error 53 us; skew 500 ppm
I20260812 01:03:36.492482 22493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:03:36 dist-test-slave-7f01 krb5kdc[21842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496616, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:03:36.495965 22493 init.cc:377] Logged in from keytab as kudu/127.21.201.67@KRBTEST.COM (short username kudu)
I20260812 01:03:36.497587 22493 webserver.cc:460] Webserver started at http://127.21.201.67:40209/ using document root <none> and password file <none>
I20260812 01:03:36.498272 22493 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:36.498417 22493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:36.498734 22493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:36.501683 22493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-2/data/instance:
uuid: "b8f6cc5644184c8f81d932659f475354"
format_stamp: "Formatted at 2026-08-12 01:03:36 on dist-test-slave-7f01"
server_key: "869af71182da2b9a266e0c6ae1d9559b"
server_key_iv: "599d219336afcd5e9b61ee809430af4a"
server_key_version: "encryptionkey@0"
I20260812 01:03:36.502527 22493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance:
uuid: "b8f6cc5644184c8f81d932659f475354"
format_stamp: "Formatted at 2026-08-12 01:03:36 on dist-test-slave-7f01"
server_key: "869af71182da2b9a266e0c6ae1d9559b"
server_key_iv: "599d219336afcd5e9b61ee809430af4a"
server_key_version: "encryptionkey@0"
I20260812 01:03:36.507694 22493 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 01:03:36.511194 22520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:36.512501 22493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:03:36.512677 22493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "b8f6cc5644184c8f81d932659f475354"
format_stamp: "Formatted at 2026-08-12 01:03:36 on dist-test-slave-7f01"
server_key: "869af71182da2b9a266e0c6ae1d9559b"
server_key_iv: "599d219336afcd5e9b61ee809430af4a"
server_key_version: "encryptionkey@0"
I20260812 01:03:36.512885 22493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:36.542222 22493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:36.546883 22493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:36.547219 22493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:36.548326 22493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:03:36.549747 22493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:03:36.549855 22493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:36.550005 22493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:03:36.550086 22493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:36.585368 22493 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:46497
I20260812 01:03:36.585414 22694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:46497 every 8 connection(s)
I20260812 01:03:36.586916 22493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 01:03:36.590059 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 22493
I20260812 01:03:36.590271 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance
I20260812 01:03:36.590776 22309 external_mini_cluster.cc:1428] Setting key acb0dd3ba8f001b00c442640cbf37fb1
Aug 12 01:03:36 dist-test-slave-7f01 krb5kdc[21842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496616, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:03:36.610410 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:36.589971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:39335 (local address 127.21.201.126:36659)
0812 01:03:36.592763 (+ 2792us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:36.592770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:36.592791 (+ 21us) server_negotiation.cc:407] Connection header received
0812 01:03:36.594244 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:36.594249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:36.594373 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:36.594496 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:36.596512 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.597402 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.599040 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.599460 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.599563 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:36.603618 (+ 4055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:36.603624 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:36.603627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:36.603662 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:36.607100 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:36.607871 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:36.607877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:36.607880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:36.608014 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:36.608455 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:36.608460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:36.608463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:36.608745 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:36.608903 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:36.609762 (+ 859us) server_negotiation.cc:299] Negotiation successful
0812 01:03:36.609999 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2666,"thread_start_us":2521,"threads_started":1}
I20260812 01:03:36.611366 22704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:36.590570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:36659 (local address 127.21.201.67:39335)
0812 01:03:36.591638 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:36.591705 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:36.593925 (+ 2220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:36.594868 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:36.594880 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:36.595542 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:03:36.596303 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:36.596325 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.597586 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.597590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:36.598881 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:36.598895 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.599695 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.599701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:36.600025 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:36.600771 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:03:36.600794 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:03:36.603380 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:03:36.607252 (+ 3872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:03:36.607261 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:03:36.607278 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:03:36.607719 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:03:36.608128 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:03:36.608133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:03:36.608136 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:03:36.608329 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:03:36.608878 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:03:36.608885 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:03:36.609365 (+ 480us) client_negotiation.cc:769] Sending connection context
0812 01:03:36.609659 (+ 294us) client_negotiation.cc:240] Negotiation successful
0812 01:03:36.610215 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":485,"threads_started":1}
I20260812 01:03:36.612829 22697 heartbeater.cc:347] Connected to a master server at 127.21.201.126:36659
I20260812 01:03:36.613157 22697 heartbeater.cc:464] Registering TS with master...
I20260812 01:03:36.613893 22697 heartbeater.cc:511] Master 127.21.201.126:36659 requested a full tablet report, sending...
I20260812 01:03:36.615455 21938 ts_manager.cc:194] Registered new tserver with Master: b8f6cc5644184c8f81d932659f475354 (127.21.201.67:46497)
I20260812 01:03:36.616467 21938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.67@KRBTEST.COM'} at 127.21.201.67:39335
I20260812 01:03:36.619617 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:03:36.639395 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:36.624026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42338 (local address 127.21.201.126:36659)
0812 01:03:36.624257 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:36.624266 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:36.625223 (+ 957us) server_negotiation.cc:407] Connection header received
0812 01:03:36.625286 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:36.625290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:36.625369 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:36.625453 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:36.627233 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.628173 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.629548 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.629971 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.630090 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:36.633200 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:36.633205 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:36.633208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:36.633241 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:36.636200 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:36.637124 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:36.637128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:36.637130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:36.637261 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:36.637809 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:36.637813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:36.637816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:36.638192 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:36.638417 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:36.638807 (+ 390us) server_negotiation.cc:299] Negotiation successful
0812 01:03:36.639032 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 01:03:36.643702 21938 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:03:36.651094 21938 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:03:36.677125 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:36.664211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:46497 (local address 127.0.0.1:32882)
0812 01:03:36.664932 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:36.664982 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:36.665153 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:36.667259 (+ 2106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:36.667268 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:36.667312 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:36.668030 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:36.668051 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.671058 (+ 3007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.671067 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:36.674291 (+ 3224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:36.674304 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.675956 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.675963 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:36.676160 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:36.676215 (+ 55us) client_negotiation.cc:769] Sending connection context
0812 01:03:36.676367 (+ 152us) client_negotiation.cc:240] Negotiation successful
0812 01:03:36.676582 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":259,"thread_start_us":325,"threads_started":1}
I20260812 01:03:36.677416 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:36.664539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32882 (local address 127.21.201.67:46497)
0812 01:03:36.666537 (+ 1998us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:36.666550 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:36.666588 (+ 38us) server_negotiation.cc:407] Connection header received
0812 01:03:36.666709 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:36.666718 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:36.667002 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:36.668317 (+ 1315us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:36.668621 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.670861 (+ 2240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.674460 (+ 3599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.675693 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.675898 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:36.676169 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:36.676424 (+ 255us) server_negotiation.cc:299] Negotiation successful
0812 01:03:36.676585 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1754,"thread_start_us":1601,"threads_started":1}
I20260812 01:03:36.678452 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:36.665151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45764 (local address 127.21.201.66:35871)
0812 01:03:36.666840 (+ 1689us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:36.666854 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:36.666883 (+ 29us) server_negotiation.cc:407] Connection header received
0812 01:03:36.666991 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:36.666999 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:36.667219 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:36.668519 (+ 1300us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:36.668842 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.670856 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.674805 (+ 3949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.676344 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.676476 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:36.676639 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:36.677921 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0812 01:03:36.678112 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1520,"thread_start_us":1377,"threads_started":1}
I20260812 01:03:36.678839 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:36.664503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:35871 (local address 127.0.0.1:45764)
0812 01:03:36.665757 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:36.665776 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:36.665889 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:36.667500 (+ 1611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:36.667508 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:36.667553 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:36.668260 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:36.668278 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.671060 (+ 2782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.671069 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:36.674575 (+ 3506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:36.674591 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.676801 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.676809 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:36.677099 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:36.677157 (+ 58us) client_negotiation.cc:769] Sending connection context
0812 01:03:36.677247 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 01:03:36.678393 (+ 1146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1126,"mutex_wait_us":1095,"thread_start_us":983,"threads_started":1}
I20260812 01:03:36.681545 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:36.667010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:35939 (local address 127.0.0.1:53760)
0812 01:03:36.673687 (+ 6677us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:36.673706 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:36.673836 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:36.674534 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:36.674541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:36.674571 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:36.674961 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:36.674974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.676875 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.676887 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:36.679136 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:36.679151 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.680820 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.680827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:36.680989 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:36.681021 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 01:03:36.681118 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 01:03:36.681242 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6553,"thread_start_us":6479,"threads_started":1}
I20260812 01:03:36.681949 22597 tablet_service.cc:1467] Processing CreateTablet for tablet bf7e66db4a5b4fdf85bf71b001182a07 (DEFAULT_TABLE table=test-table [id=b16572957a2e407a8d1f377342b6c420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:03:36.682643 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:36.667117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53760 (local address 127.21.201.65:35939)
0812 01:03:36.672763 (+ 5646us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:36.672770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:36.673973 (+ 1203us) server_negotiation.cc:407] Connection header received
0812 01:03:36.674096 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:36.674104 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:36.674347 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:36.674535 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:36.675128 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.676690 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.679320 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.680571 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.680729 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:36.680983 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:36.681184 (+ 201us) server_negotiation.cc:299] Negotiation successful
0812 01:03:36.682324 (+ 1140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5478,"thread_start_us":5340,"threads_started":1}
I20260812 01:03:36.682778 22377 tablet_service.cc:1467] Processing CreateTablet for tablet bf7e66db4a5b4fdf85bf71b001182a07 (DEFAULT_TABLE table=test-table [id=b16572957a2e407a8d1f377342b6c420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:03:36.683518 22597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf7e66db4a5b4fdf85bf71b001182a07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:03:36.684193 22377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf7e66db4a5b4fdf85bf71b001182a07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:03:36.686122 22147 tablet_service.cc:1467] Processing CreateTablet for tablet bf7e66db4a5b4fdf85bf71b001182a07 (DEFAULT_TABLE table=test-table [id=b16572957a2e407a8d1f377342b6c420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:03:36.687443 22147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf7e66db4a5b4fdf85bf71b001182a07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:03:36.696126 22731 tablet_bootstrap.cc:492] T bf7e66db4a5b4fdf85bf71b001182a07 P 332a52b29b1c4c329dc16a1877de4c81: Bootstrap starting.
I20260812 01:03:36.699661 22732 tablet_bootstrap.cc:492] T bf7e66db4a5b4fdf85bf71b001182a07 P b8f6cc5644184c8f81d932659f475354: Bootstrap starting.
I20260812 01:03:36.700203 22731 tablet_bootstrap.cc:654] T bf7e66db4a5b4fdf85bf71b001182a07 P 332a52b29b1c4c329dc16a1877de4c81: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:36.701634 22731 log.cc:824] T bf7e66db4a5b4fdf85bf71b001182a07 P 332a52b29b1c4c329dc16a1877de4c81: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:36.702098 22736 tablet_bootstrap.cc:492] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b: Bootstrap starting.
I20260812 01:03:36.703531 22732 tablet_bootstrap.cc:654] T bf7e66db4a5b4fdf85bf71b001182a07 P b8f6cc5644184c8f81d932659f475354: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:36.704838 22732 log.cc:824] T bf7e66db4a5b4fdf85bf71b001182a07 P b8f6cc5644184c8f81d932659f475354: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:36.705302 22736 tablet_bootstrap.cc:654] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:36.705878 22731 tablet_bootstrap.cc:492] T bf7e66db4a5b4fdf85bf71b001182a07 P 332a52b29b1c4c329dc16a1877de4c81: No bootstrap required, opened a new log
I20260812 01:03:36.706300 22731 ts_tablet_manager.cc:1397] T bf7e66db4a5b4fdf85bf71b001182a07 P 332a52b29b1c4c329dc16a1877de4c81: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260812 01:03:36.706640 22736 log.cc:824] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:36.708235 22732 tablet_bootstrap.cc:492] T bf7e66db4a5b4fdf85bf71b001182a07 P b8f6cc5644184c8f81d932659f475354: No bootstrap required, opened a new log
I20260812 01:03:36.708503 22732 ts_tablet_manager.cc:1397] T bf7e66db4a5b4fdf85bf71b001182a07 P b8f6cc5644184c8f81d932659f475354: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260812 01:03:36.710392 22736 tablet_bootstrap.cc:492] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b: No bootstrap required, opened a new log
I20260812 01:03:36.710649 22736 ts_tablet_manager.cc:1397] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260812 01:03:36.714318 22731 raft_consensus.cc:348] T bf7e66db4a5b4fdf85bf71b001182a07 P 332a52b29b1c4c329dc16a1877de4c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f6cc5644184c8f81d932659f475354" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 46497 } } peers { permanent_uuid: "332a52b29b1c4c329dc16a1877de4c81" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35871 } } peers { permanent_uuid: "94abb9efef014df0a0df400785c5893b" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35939 } }
I20260812 01:03:36.714721 22731 raft_consensus.cc:374] T bf7e66db4a5b4fdf85bf71b001182a07 P 332a52b29b1c4c329dc16a1877de4c81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:36.714864 22731 raft_consensus.cc:729] T bf7e66db4a5b4fdf85bf71b001182a07 P 332a52b29b1c4c329dc16a1877de4c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 332a52b29b1c4c329dc16a1877de4c81, State: Initialized, Role: FOLLOWER
I20260812 01:03:36.715499 22731 consensus_queue.cc:260] T bf7e66db4a5b4fdf85bf71b001182a07 P 332a52b29b1c4c329dc16a1877de4c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f6cc5644184c8f81d932659f475354" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 46497 } } peers { permanent_uuid: "332a52b29b1c4c329dc16a1877de4c81" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35871 } } peers { permanent_uuid: "94abb9efef014df0a0df400785c5893b" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35939 } }
I20260812 01:03:36.716719 22481 heartbeater.cc:503] Master 127.21.201.126:36659 was elected leader, sending a full tablet report...
I20260812 01:03:36.719444 22732 raft_consensus.cc:348] T bf7e66db4a5b4fdf85bf71b001182a07 P b8f6cc5644184c8f81d932659f475354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f6cc5644184c8f81d932659f475354" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 46497 } } peers { permanent_uuid: "332a52b29b1c4c329dc16a1877de4c81" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35871 } } peers { permanent_uuid: "94abb9efef014df0a0df400785c5893b" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35939 } }
I20260812 01:03:36.719914 22732 raft_consensus.cc:374] T bf7e66db4a5b4fdf85bf71b001182a07 P b8f6cc5644184c8f81d932659f475354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:36.720021 22732 raft_consensus.cc:729] T bf7e66db4a5b4fdf85bf71b001182a07 P b8f6cc5644184c8f81d932659f475354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f6cc5644184c8f81d932659f475354, State: Initialized, Role: FOLLOWER
I20260812 01:03:36.720623 22732 consensus_queue.cc:260] T bf7e66db4a5b4fdf85bf71b001182a07 P b8f6cc5644184c8f81d932659f475354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f6cc5644184c8f81d932659f475354" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 46497 } } peers { permanent_uuid: "332a52b29b1c4c329dc16a1877de4c81" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35871 } } peers { permanent_uuid: "94abb9efef014df0a0df400785c5893b" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35939 } }
I20260812 01:03:36.721113 22731 ts_tablet_manager.cc:1428] T bf7e66db4a5b4fdf85bf71b001182a07 P 332a52b29b1c4c329dc16a1877de4c81: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260812 01:03:36.722147 22697 heartbeater.cc:503] Master 127.21.201.126:36659 was elected leader, sending a full tablet report...
I20260812 01:03:36.721880 22736 raft_consensus.cc:348] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f6cc5644184c8f81d932659f475354" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 46497 } } peers { permanent_uuid: "332a52b29b1c4c329dc16a1877de4c81" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35871 } } peers { permanent_uuid: "94abb9efef014df0a0df400785c5893b" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35939 } }
I20260812 01:03:36.722319 22736 raft_consensus.cc:374] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:36.722424 22736 raft_consensus.cc:729] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94abb9efef014df0a0df400785c5893b, State: Initialized, Role: FOLLOWER
I20260812 01:03:36.722856 22732 ts_tablet_manager.cc:1428] T bf7e66db4a5b4fdf85bf71b001182a07 P b8f6cc5644184c8f81d932659f475354: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260812 01:03:36.723110 22736 consensus_queue.cc:260] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f6cc5644184c8f81d932659f475354" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 46497 } } peers { permanent_uuid: "332a52b29b1c4c329dc16a1877de4c81" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35871 } } peers { permanent_uuid: "94abb9efef014df0a0df400785c5893b" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35939 } }
I20260812 01:03:36.728992 22736 ts_tablet_manager.cc:1428] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b: Time spent starting tablet: real 0.018s user 0.014s sys 0.005s
W20260812 01:03:36.775416 22483 tablet.cc:2369] T bf7e66db4a5b4fdf85bf71b001182a07 P 332a52b29b1c4c329dc16a1877de4c81: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:03:36.839037 22698 tablet.cc:2369] T bf7e66db4a5b4fdf85bf71b001182a07 P b8f6cc5644184c8f81d932659f475354: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:03:36.943912 22252 tablet.cc:2369] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:03:36.981055 22743 raft_consensus.cc:482] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:03:36.981357 22743 raft_consensus.cc:504] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f6cc5644184c8f81d932659f475354" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 46497 } } peers { permanent_uuid: "332a52b29b1c4c329dc16a1877de4c81" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35871 } } peers { permanent_uuid: "94abb9efef014df0a0df400785c5893b" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35939 } }
I20260812 01:03:36.983198 22743 leader_election.cc:288] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f6cc5644184c8f81d932659f475354 (127.21.201.67:46497), 332a52b29b1c4c329dc16a1877de4c81 (127.21.201.66:35871)
I20260812 01:03:36.993194 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:36.983862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:40495 (local address 127.21.201.67:46497)
0812 01:03:36.984112 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:36.984118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:36.984986 (+ 868us) server_negotiation.cc:407] Connection header received
0812 01:03:36.985168 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:36.985172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:36.985250 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:36.985344 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:36.986212 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.987415 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.990680 (+ 3265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.992005 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.992207 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:36.992273 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:36.992719 (+ 446us) server_negotiation.cc:299] Negotiation successful
0812 01:03:36.992865 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 01:03:36.993450 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:36.983686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:46497 (local address 127.21.201.65:40495)
0812 01:03:36.984888 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:36.984904 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:36.985011 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:36.985387 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:36.985392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:36.985490 (+ 98us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:36.986018 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:36.986027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.987588 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.987592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:36.990470 (+ 2878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:36.990487 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.992286 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.992294 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:36.992516 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:36.992599 (+ 83us) client_negotiation.cc:769] Sending connection context
0812 01:03:36.992683 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 01:03:36.992913 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1084,"mutex_wait_us":348,"spinlock_wait_cycles":22912,"thread_start_us":611,"threads_started":1}
I20260812 01:03:36.993485 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:36.983900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:57877 (local address 127.21.201.66:35871)
0812 01:03:36.984236 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:36.984244 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:36.984880 (+ 636us) server_negotiation.cc:407] Connection header received
0812 01:03:36.985130 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:36.985135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:36.985223 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:36.985345 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:36.986212 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.987408 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.990680 (+ 3272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.992030 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.992190 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:36.992272 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:36.992719 (+ 447us) server_negotiation.cc:299] Negotiation successful
0812 01:03:36.992892 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260812 01:03:36.993502 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:36.983543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:35871 (local address 127.21.201.65:57877)
0812 01:03:36.984747 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:36.984774 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:36.984956 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:36.985387 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:36.985392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:36.985423 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:36.985968 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:36.985988 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.987586 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.987592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:36.990481 (+ 2889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:36.990490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:36.992304 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:36.992311 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:36.992505 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:36.992575 (+ 70us) client_negotiation.cc:769] Sending connection context
0812 01:03:36.992683 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 01:03:36.992913 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1005,"mutex_wait_us":232,"thread_start_us":1055,"threads_started":1}
I20260812 01:03:36.994843 22407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7e66db4a5b4fdf85bf71b001182a07" candidate_uuid: "94abb9efef014df0a0df400785c5893b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "332a52b29b1c4c329dc16a1877de4c81" is_pre_election: true
I20260812 01:03:36.994851 22632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7e66db4a5b4fdf85bf71b001182a07" candidate_uuid: "94abb9efef014df0a0df400785c5893b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f6cc5644184c8f81d932659f475354" is_pre_election: true
I20260812 01:03:36.995405 22632 raft_consensus.cc:2454] T bf7e66db4a5b4fdf85bf71b001182a07 P b8f6cc5644184c8f81d932659f475354 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94abb9efef014df0a0df400785c5893b in term 0.
I20260812 01:03:36.995415 22407 raft_consensus.cc:2454] T bf7e66db4a5b4fdf85bf71b001182a07 P 332a52b29b1c4c329dc16a1877de4c81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94abb9efef014df0a0df400785c5893b in term 0.
I20260812 01:03:36.996162 22066 leader_election.cc:302] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 332a52b29b1c4c329dc16a1877de4c81, 94abb9efef014df0a0df400785c5893b, b8f6cc5644184c8f81d932659f475354; no voters:
I20260812 01:03:36.996635 22743 raft_consensus.cc:2781] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:03:36.996826 22743 raft_consensus.cc:482] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:03:36.996942 22743 raft_consensus.cc:3037] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:36.999356 22743 raft_consensus.cc:504] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f6cc5644184c8f81d932659f475354" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 46497 } } peers { permanent_uuid: "332a52b29b1c4c329dc16a1877de4c81" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35871 } } peers { permanent_uuid: "94abb9efef014df0a0df400785c5893b" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35939 } }
I20260812 01:03:37.000077 22743 leader_election.cc:288] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b [CANDIDATE]: Term 1 election: Requested vote from peers b8f6cc5644184c8f81d932659f475354 (127.21.201.67:46497), 332a52b29b1c4c329dc16a1877de4c81 (127.21.201.66:35871)
I20260812 01:03:37.000528 22632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7e66db4a5b4fdf85bf71b001182a07" candidate_uuid: "94abb9efef014df0a0df400785c5893b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f6cc5644184c8f81d932659f475354"
I20260812 01:03:37.000610 22407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7e66db4a5b4fdf85bf71b001182a07" candidate_uuid: "94abb9efef014df0a0df400785c5893b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "332a52b29b1c4c329dc16a1877de4c81"
I20260812 01:03:37.000782 22632 raft_consensus.cc:3037] T bf7e66db4a5b4fdf85bf71b001182a07 P b8f6cc5644184c8f81d932659f475354 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:37.000862 22407 raft_consensus.cc:3037] T bf7e66db4a5b4fdf85bf71b001182a07 P 332a52b29b1c4c329dc16a1877de4c81 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:37.003366 22632 raft_consensus.cc:2454] T bf7e66db4a5b4fdf85bf71b001182a07 P b8f6cc5644184c8f81d932659f475354 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94abb9efef014df0a0df400785c5893b in term 1.
I20260812 01:03:37.003793 22407 raft_consensus.cc:2454] T bf7e66db4a5b4fdf85bf71b001182a07 P 332a52b29b1c4c329dc16a1877de4c81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94abb9efef014df0a0df400785c5893b in term 1.
I20260812 01:03:37.003926 22065 leader_election.cc:302] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94abb9efef014df0a0df400785c5893b, b8f6cc5644184c8f81d932659f475354; no voters:
I20260812 01:03:37.004259 22743 raft_consensus.cc:2781] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:03:37.004750 22743 raft_consensus.cc:686] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b [term 1 LEADER]: Becoming Leader. State: Replica: 94abb9efef014df0a0df400785c5893b, State: Running, Role: LEADER
I20260812 01:03:37.005358 22743 consensus_queue.cc:237] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f6cc5644184c8f81d932659f475354" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 46497 } } peers { permanent_uuid: "332a52b29b1c4c329dc16a1877de4c81" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35871 } } peers { permanent_uuid: "94abb9efef014df0a0df400785c5893b" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35939 } }
I20260812 01:03:37.010273 21936 catalog_manager.cc:5495] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b reported cstate change: term changed from 0 to 1, leader changed from <none> to 94abb9efef014df0a0df400785c5893b (127.21.201.65). New cstate: current_term: 1 leader_uuid: "94abb9efef014df0a0df400785c5893b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f6cc5644184c8f81d932659f475354" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 46497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "332a52b29b1c4c329dc16a1877de4c81" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94abb9efef014df0a0df400785c5893b" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35939 } health_report { overall_health: HEALTHY } } }
I20260812 01:03:37.047899 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:37.041919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53776 (local address 127.21.201.65:35939)
0812 01:03:37.042152 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:37.042159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:37.042268 (+ 109us) server_negotiation.cc:407] Connection header received
0812 01:03:37.042534 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:37.042538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:37.042629 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:37.042802 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:03:37.043254 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:37.044064 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:37.045486 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:37.045923 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:37.046086 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:37.046722 (+ 636us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:03:37.047210 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:03:37.047415 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:37.047530 (+ 115us) server_negotiation.cc:299] Negotiation successful
0812 01:03:37.047591 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 01:03:37.056600 22407 raft_consensus.cc:1261] T bf7e66db4a5b4fdf85bf71b001182a07 P 332a52b29b1c4c329dc16a1877de4c81 [term 1 FOLLOWER]: Refusing update from remote peer 94abb9efef014df0a0df400785c5893b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:03:37.056600 22632 raft_consensus.cc:1261] T bf7e66db4a5b4fdf85bf71b001182a07 P b8f6cc5644184c8f81d932659f475354 [term 1 FOLLOWER]: Refusing update from remote peer 94abb9efef014df0a0df400785c5893b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:03:37.057799 22748 consensus_queue.cc:1035] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f6cc5644184c8f81d932659f475354" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 46497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:03:37.058209 22743 consensus_queue.cc:1035] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b [LEADER]: Connected to new peer: Peer: permanent_uuid: "332a52b29b1c4c329dc16a1877de4c81" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:03:37.072095 22752 mvcc.cc:205] Tried to move back new op lower bound from 7317490143451697152 to 7317490143258148864. Current Snapshot: MvccSnapshot[applied={T|T < 7317490143451697152}]
I20260812 01:03:37.088250 21938 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:03:37.088600 21938 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:03:37.092932 21938 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f21c73e37ae4f4d9be848b1261c2aca: Sending DeleteTablet for 3 replicas of tablet bf7e66db4a5b4fdf85bf71b001182a07
I20260812 01:03:37.094319 22597 tablet_service.cc:1514] Processing DeleteTablet for tablet bf7e66db4a5b4fdf85bf71b001182a07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:03:37 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:32882
I20260812 01:03:37.094316 22147 tablet_service.cc:1514] Processing DeleteTablet for tablet bf7e66db4a5b4fdf85bf71b001182a07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:03:37 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:53760
I20260812 01:03:37.094424 22377 tablet_service.cc:1514] Processing DeleteTablet for tablet bf7e66db4a5b4fdf85bf71b001182a07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:03:37 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:45764
I20260812 01:03:37.096941 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 22042
I20260812 01:03:37.106853 22779 tablet_replica.cc:330] T bf7e66db4a5b4fdf85bf71b001182a07 P 332a52b29b1c4c329dc16a1877de4c81: stopping tablet replica
I20260812 01:03:37.107577 22779 raft_consensus.cc:2229] T bf7e66db4a5b4fdf85bf71b001182a07 P 332a52b29b1c4c329dc16a1877de4c81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:03:37.108340 22779 raft_consensus.cc:2258] T bf7e66db4a5b4fdf85bf71b001182a07 P 332a52b29b1c4c329dc16a1877de4c81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:03:37.110893 22779 ts_tablet_manager.cc:1905] T bf7e66db4a5b4fdf85bf71b001182a07 P 332a52b29b1c4c329dc16a1877de4c81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:03:37.115228 22778 tablet_replica.cc:330] T bf7e66db4a5b4fdf85bf71b001182a07 P b8f6cc5644184c8f81d932659f475354: stopping tablet replica
I20260812 01:03:37.115903 22778 raft_consensus.cc:2229] T bf7e66db4a5b4fdf85bf71b001182a07 P b8f6cc5644184c8f81d932659f475354 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:03:37.116586 22778 raft_consensus.cc:2258] T bf7e66db4a5b4fdf85bf71b001182a07 P b8f6cc5644184c8f81d932659f475354 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:03:37.117962 22775 tablet_replica.cc:330] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b: stopping tablet replica
I20260812 01:03:37.118468 22775 raft_consensus.cc:2229] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:03:37.118983 22778 ts_tablet_manager.cc:1905] T bf7e66db4a5b4fdf85bf71b001182a07 P b8f6cc5644184c8f81d932659f475354: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:03:37.118955 22779 ts_tablet_manager.cc:1918] T bf7e66db4a5b4fdf85bf71b001182a07 P 332a52b29b1c4c329dc16a1877de4c81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:03:37.119145 22779 log.cc:1196] T bf7e66db4a5b4fdf85bf71b001182a07 P 332a52b29b1c4c329dc16a1877de4c81: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/bf7e66db4a5b4fdf85bf71b001182a07
I20260812 01:03:37.119340 22775 raft_consensus.cc:2258] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:03:37.119834 22779 ts_tablet_manager.cc:1939] T bf7e66db4a5b4fdf85bf71b001182a07 P 332a52b29b1c4c329dc16a1877de4c81: Deleting consensus metadata
I20260812 01:03:37.121619 22775 ts_tablet_manager.cc:1905] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:03:37.121954 21901 catalog_manager.cc:4841] TS 332a52b29b1c4c329dc16a1877de4c81 (127.21.201.66:35871): tablet bf7e66db4a5b4fdf85bf71b001182a07 (table test-table [id=b16572957a2e407a8d1f377342b6c420]) successfully deleted
Aug 12 01:03:37 dist-test-slave-7f01 krb5kdc[21842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.65@KRBTEST.COM
I20260812 01:03:37.127200 22778 ts_tablet_manager.cc:1918] T bf7e66db4a5b4fdf85bf71b001182a07 P b8f6cc5644184c8f81d932659f475354: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:03:37.127499 22778 log.cc:1196] T bf7e66db4a5b4fdf85bf71b001182a07 P b8f6cc5644184c8f81d932659f475354: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/bf7e66db4a5b4fdf85bf71b001182a07
I20260812 01:03:37.128157 22778 ts_tablet_manager.cc:1939] T bf7e66db4a5b4fdf85bf71b001182a07 P b8f6cc5644184c8f81d932659f475354: Deleting consensus metadata
I20260812 01:03:37.130538 22775 ts_tablet_manager.cc:1918] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:03:37.130824 22775 log.cc:1196] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786496278741276-22309-0/minicluster-data/ts-0/wal/wals/bf7e66db4a5b4fdf85bf71b001182a07
I20260812 01:03:37.131023 21899 catalog_manager.cc:4841] TS b8f6cc5644184c8f81d932659f475354 (127.21.201.67:46497): tablet bf7e66db4a5b4fdf85bf71b001182a07 (table test-table [id=b16572957a2e407a8d1f377342b6c420]) successfully deleted
I20260812 01:03:37.131601 22775 ts_tablet_manager.cc:1939] T bf7e66db4a5b4fdf85bf71b001182a07 P 94abb9efef014df0a0df400785c5893b: Deleting consensus metadata
I20260812 01:03:37.133261 21907 catalog_manager.cc:4841] TS 94abb9efef014df0a0df400785c5893b (127.21.201.65:35939): tablet bf7e66db4a5b4fdf85bf71b001182a07 (table test-table [id=b16572957a2e407a8d1f377342b6c420]) successfully deleted
I20260812 01:03:37.136054 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:37.098776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53790 (local address 127.21.201.65:35939)
0812 01:03:37.099034 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:37.099043 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:37.102798 (+ 3755us) server_negotiation.cc:407] Connection header received
0812 01:03:37.103717 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:37.103727 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:37.103874 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:37.103972 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:37.108427 (+ 4455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:37.110060 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:37.114962 (+ 4902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:37.115661 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:37.115825 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:37.127985 (+ 12160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:37.128001 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:37.128023 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:37.128072 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:37.132772 (+ 4700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:37.133813 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:37.133821 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:37.133823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:37.134006 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:37.134459 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:37.134462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:37.134464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:37.134827 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:37.135065 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:37.135455 (+ 390us) server_negotiation.cc:299] Negotiation successful
0812 01:03:37.135647 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 01:03:37.136791 22241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:37.241976 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 22042
I20260812 01:03:37.272107 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 22258
Aug 12 01:03:37 dist-test-slave-7f01 krb5kdc[21842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.66@KRBTEST.COM
I20260812 01:03:37.288017 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:37.272947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54846 (local address 127.21.201.66:35871)
0812 01:03:37.273213 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:37.273218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:37.273242 (+ 24us) server_negotiation.cc:407] Connection header received
0812 01:03:37.273441 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:37.273445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:37.273513 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:37.273603 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:37.274973 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:37.276011 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:37.277306 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:37.277709 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:37.277802 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:37.282172 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:37.282187 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:37.282202 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:37.282237 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:37.285215 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:37.285868 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:37.285875 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:37.285878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:37.286010 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:37.286356 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:37.286360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:37.286361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:37.286734 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:37.286985 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:37.287344 (+ 359us) server_negotiation.cc:299] Negotiation successful
0812 01:03:37.287520 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260812 01:03:37.288863 22475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:37.382272 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 22258
I20260812 01:03:37.408279 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 22493
Aug 12 01:03:37 dist-test-slave-7f01 krb5kdc[21842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.67@KRBTEST.COM
I20260812 01:03:37.424559 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:37.409164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58308 (local address 127.21.201.67:46497)
0812 01:03:37.409344 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:37.409349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:37.409374 (+ 25us) server_negotiation.cc:407] Connection header received
0812 01:03:37.409558 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:37.409562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:37.409636 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:37.409739 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:37.411207 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:37.412082 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:37.413501 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:37.413954 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:37.414071 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:37.418985 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:37.419001 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:37.419018 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:37.419055 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:37.421818 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:37.422517 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:37.422524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:37.422526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:37.422673 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:37.423043 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:37.423047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:37.423049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:37.423429 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:37.423674 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:37.424010 (+ 336us) server_negotiation.cc:299] Negotiation successful
0812 01:03:37.424184 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 01:03:37.425330 22688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:37.516991 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 22493
I20260812 01:03:37.539171 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 21865
I20260812 01:03:37.540477 21983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:37.661132 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 21865
2026-08-12T01:03:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (5305 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-12T01:03:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:03:37Z Disabled control of system clock
I20260812 01:03:37.764914 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:46519
--webserver_interface=127.21.201.126
--webserver_port=0
--builtin_ntp_servers=127.21.201.84:41979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:46519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 01:03:38.121346 22793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:38.121666 22793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:38.121755 22793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:38.129257 22793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:03:38.129377 22793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:38.129417 22793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:38.129477 22793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:03:38.129523 22793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:03:38.139328 22793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:41979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:46519
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:46519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:38.140672 22793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:38.142503 22793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:38.154917 22810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:38.155052 22802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:38.155032 22804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:38.155790 22793 server_base.cc:1034] running on GCE node
I20260812 01:03:38.156606 22793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:38.158032 22793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:38.159250 22793 hybrid_clock.cc:648] HybridClock initialized: now 1786496618159218 us; error 46 us; skew 500 ppm
I20260812 01:03:38.159577 22793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:03:38.162037 22793 webserver.cc:460] Webserver started at http://127.21.201.126:40567/ using document root <none> and password file <none>
I20260812 01:03:38.162704 22793 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:38.162797 22793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:38.163075 22793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:38.166059 22793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "905879d6449448a8b3b1a8327bc37095"
format_stamp: "Formatted at 2026-08-12 01:03:38 on dist-test-slave-7f01"
server_key: "f6f1cd13e80e1ade05e3d68556c4196f"
server_key_iv: "a23150f0fc5d62e0ab2a0692d7c1406b"
server_key_version: "encryptionkey@0"
I20260812 01:03:38.166886 22793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "905879d6449448a8b3b1a8327bc37095"
format_stamp: "Formatted at 2026-08-12 01:03:38 on dist-test-slave-7f01"
server_key: "f6f1cd13e80e1ade05e3d68556c4196f"
server_key_iv: "a23150f0fc5d62e0ab2a0692d7c1406b"
server_key_version: "encryptionkey@0"
I20260812 01:03:38.172256 22793 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260812 01:03:38.175853 22819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:38.177343 22793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:03:38.177579 22793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "905879d6449448a8b3b1a8327bc37095"
format_stamp: "Formatted at 2026-08-12 01:03:38 on dist-test-slave-7f01"
server_key: "f6f1cd13e80e1ade05e3d68556c4196f"
server_key_iv: "a23150f0fc5d62e0ab2a0692d7c1406b"
server_key_version: "encryptionkey@0"
I20260812 01:03:38.177831 22793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:38.191632 22793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:38.192513 22793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:38.192821 22793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:38.213111 22793 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:46519
I20260812 01:03:38.213146 22900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:46519 every 8 connection(s)
I20260812 01:03:38.214536 22793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:03:38.215858 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 22793
I20260812 01:03:38.216277 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:03:38.216769 22309 external_mini_cluster.cc:1428] Setting key dcdbe739c22430f42fc9fcaf7cee3345
I20260812 01:03:38.219271 22901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:03:38.232074 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:38.218853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50522 (local address 127.21.201.126:46519)
0812 01:03:38.219941 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:38.219952 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:38.219995 (+ 43us) server_negotiation.cc:407] Connection header received
0812 01:03:38.221673 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:38.221694 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:38.222147 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:38.222473 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:38.223290 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:38.224790 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:38.226835 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:38.227966 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:38.228149 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:38.229531 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:38.229539 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:38.229552 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:38.230136 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:38.230218 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:38.230244 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:38.230338 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:38.230588 (+ 250us) server_negotiation.cc:299] Negotiation successful
0812 01:03:38.230969 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":402,"threads_started":1}
I20260812 01:03:38.232867 22901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 905879d6449448a8b3b1a8327bc37095: Bootstrap starting.
I20260812 01:03:38.236719 22901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 905879d6449448a8b3b1a8327bc37095: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:38.237777 22901 log.cc:824] T 00000000000000000000000000000000 P 905879d6449448a8b3b1a8327bc37095: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:38.240764 22901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 905879d6449448a8b3b1a8327bc37095: No bootstrap required, opened a new log
I20260812 01:03:38.248286 22901 raft_consensus.cc:348] T 00000000000000000000000000000000 P 905879d6449448a8b3b1a8327bc37095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905879d6449448a8b3b1a8327bc37095" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46519 } }
I20260812 01:03:38.248620 22901 raft_consensus.cc:374] T 00000000000000000000000000000000 P 905879d6449448a8b3b1a8327bc37095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:38.248694 22901 raft_consensus.cc:729] T 00000000000000000000000000000000 P 905879d6449448a8b3b1a8327bc37095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 905879d6449448a8b3b1a8327bc37095, State: Initialized, Role: FOLLOWER
I20260812 01:03:38.249190 22901 consensus_queue.cc:260] T 00000000000000000000000000000000 P 905879d6449448a8b3b1a8327bc37095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905879d6449448a8b3b1a8327bc37095" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46519 } }
I20260812 01:03:38.249413 22901 raft_consensus.cc:388] T 00000000000000000000000000000000 P 905879d6449448a8b3b1a8327bc37095 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:03:38.249521 22901 raft_consensus.cc:482] T 00000000000000000000000000000000 P 905879d6449448a8b3b1a8327bc37095 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:03:38.249650 22901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 905879d6449448a8b3b1a8327bc37095 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:38.252211 22901 raft_consensus.cc:504] T 00000000000000000000000000000000 P 905879d6449448a8b3b1a8327bc37095 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905879d6449448a8b3b1a8327bc37095" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46519 } }
I20260812 01:03:38.252665 22901 leader_election.cc:302] T 00000000000000000000000000000000 P 905879d6449448a8b3b1a8327bc37095 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 905879d6449448a8b3b1a8327bc37095; no voters:
I20260812 01:03:38.253191 22901 leader_election.cc:288] T 00000000000000000000000000000000 P 905879d6449448a8b3b1a8327bc37095 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:03:38.253353 22909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 905879d6449448a8b3b1a8327bc37095 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:03:38.254132 22909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 905879d6449448a8b3b1a8327bc37095 [term 1 LEADER]: Becoming Leader. State: Replica: 905879d6449448a8b3b1a8327bc37095, State: Running, Role: LEADER
I20260812 01:03:38.254669 22909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 905879d6449448a8b3b1a8327bc37095 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905879d6449448a8b3b1a8327bc37095" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46519 } }
I20260812 01:03:38.255281 22901 sys_catalog.cc:564] T 00000000000000000000000000000000 P 905879d6449448a8b3b1a8327bc37095 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:03:38.257642 22912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 905879d6449448a8b3b1a8327bc37095 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 905879d6449448a8b3b1a8327bc37095. Latest consensus state: current_term: 1 leader_uuid: "905879d6449448a8b3b1a8327bc37095" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905879d6449448a8b3b1a8327bc37095" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46519 } } }
I20260812 01:03:38.258054 22912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 905879d6449448a8b3b1a8327bc37095 [sys.catalog]: This master's current role is: LEADER
I20260812 01:03:38.258517 22911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 905879d6449448a8b3b1a8327bc37095 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "905879d6449448a8b3b1a8327bc37095" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905879d6449448a8b3b1a8327bc37095" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46519 } } }
I20260812 01:03:38.258870 22911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 905879d6449448a8b3b1a8327bc37095 [sys.catalog]: This master's current role is: LEADER
I20260812 01:03:38.260862 22921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:03:38.266551 22921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:03:38.276660 22921 catalog_manager.cc:1347] Generated new cluster ID: 430206a148b040489fdb244e518c807e
I20260812 01:03:38.276787 22921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:03:38.303263 22921 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:03:38.305491 22921 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:03:38.323796 22921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 905879d6449448a8b3b1a8327bc37095: Generated new TSK 0
I20260812 01:03:38.325050 22921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:03:38.335412 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:0
--local_ip_for_outbound_sockets=127.21.201.65
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:46519
--builtin_ntp_servers=127.21.201.84:41979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 01:03:38.717207 22939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:38.717492 22939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:38.717587 22939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:38.724979 22939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:38.725090 22939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:38.725205 22939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 01:03:38.734284 22939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:41979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:46519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:38.735592 22939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:38.737371 22939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:38.752136 22945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:38.752044 22947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:38.752091 22951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:38.753350 22939 server_base.cc:1034] running on GCE node
I20260812 01:03:38.753994 22939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:38.755095 22939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:38.756332 22939 hybrid_clock.cc:648] HybridClock initialized: now 1786496618756282 us; error 35 us; skew 500 ppm
I20260812 01:03:38.756665 22939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:03:38.759052 22939 webserver.cc:460] Webserver started at http://127.21.201.65:43389/ using document root <none> and password file <none>
I20260812 01:03:38.759883 22939 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:38.760038 22939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:38.760434 22939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:38.763850 22939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-0/data/instance:
uuid: "fc9d3bf3f0804bf1bc7b274416f7ebad"
format_stamp: "Formatted at 2026-08-12 01:03:38 on dist-test-slave-7f01"
server_key: "1b7c6098ca30f26ff04032a611df700c"
server_key_iv: "ddd2c385ca37894c0138d537068f1a4c"
server_key_version: "encryptionkey@0"
I20260812 01:03:38.764693 22939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance:
uuid: "fc9d3bf3f0804bf1bc7b274416f7ebad"
format_stamp: "Formatted at 2026-08-12 01:03:38 on dist-test-slave-7f01"
server_key: "1b7c6098ca30f26ff04032a611df700c"
server_key_iv: "ddd2c385ca37894c0138d537068f1a4c"
server_key_version: "encryptionkey@0"
I20260812 01:03:38.770238 22939 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:03:38.773945 22964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:38.775226 22939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 01:03:38.775419 22939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "fc9d3bf3f0804bf1bc7b274416f7ebad"
format_stamp: "Formatted at 2026-08-12 01:03:38 on dist-test-slave-7f01"
server_key: "1b7c6098ca30f26ff04032a611df700c"
server_key_iv: "ddd2c385ca37894c0138d537068f1a4c"
server_key_version: "encryptionkey@0"
I20260812 01:03:38.775664 22939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:38.792984 22939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:38.793890 22939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:38.794245 22939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:38.795298 22939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:03:38.797097 22939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:03:38.797231 22939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:38.797371 22939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:03:38.797464 22939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:38.835892 22939 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:46011
I20260812 01:03:38.835986 23163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:46011 every 8 connection(s)
I20260812 01:03:38.837313 22939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 01:03:38.844626 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 22939
I20260812 01:03:38.844828 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance
I20260812 01:03:38.845376 22309 external_mini_cluster.cc:1428] Setting key 31564ab2e01ad845da6a188c3bf55a26
I20260812 01:03:38.848312 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:0
--local_ip_for_outbound_sockets=127.21.201.66
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:46519
--builtin_ntp_servers=127.21.201.84:41979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 01:03:38.850636 23169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:38.840154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:44661 (local address 127.21.201.126:46519)
0812 01:03:38.840650 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:38.840655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:38.841724 (+ 1069us) server_negotiation.cc:407] Connection header received
0812 01:03:38.843561 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:38.843567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:38.843662 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:38.843801 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:38.845183 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:38.846042 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:38.847677 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:38.848147 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:38.848295 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:38.849740 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:38.849745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:38.849748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:38.849765 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:38.849807 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:38.849820 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:38.849917 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:38.850161 (+ 244us) server_negotiation.cc:299] Negotiation successful
0812 01:03:38.850265 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":269,"threads_started":1}
I20260812 01:03:38.851629 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:38.840673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46519 (local address 127.21.201.65:44661)
0812 01:03:38.841571 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:38.841608 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:38.843284 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:38.844036 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:38.844044 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:38.844133 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:03:38.844967 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:38.844997 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:38.846236 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:38.846249 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:38.847510 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:38.847524 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:38.848388 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:38.848395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:38.848682 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:38.849417 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:03:38.849420 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:03:38.849457 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:03:38.849540 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:03:38.849931 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:03:38.849944 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:03:38.849959 (+ 15us) client_negotiation.cc:769] Sending connection context
0812 01:03:38.850147 (+ 188us) client_negotiation.cc:240] Negotiation successful
0812 01:03:38.850528 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":283,"threads_started":1}
I20260812 01:03:38.853304 23165 heartbeater.cc:347] Connected to a master server at 127.21.201.126:46519
I20260812 01:03:38.853633 23165 heartbeater.cc:464] Registering TS with master...
I20260812 01:03:38.854389 23165 heartbeater.cc:511] Master 127.21.201.126:46519 requested a full tablet report, sending...
I20260812 01:03:38.856762 22848 ts_manager.cc:194] Registered new tserver with Master: fc9d3bf3f0804bf1bc7b274416f7ebad (127.21.201.65:46011)
I20260812 01:03:38.858954 22848 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.201.65:44661
W20260812 01:03:39.195355 23172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:39.195690 23172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:39.195873 23172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:39.203181 23172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:39.203310 23172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:39.203485 23172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 01:03:39.212697 23172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:41979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:46519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:39.213937 23172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:39.215718 23172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:39.230790 23183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:39.231029 23185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:39.231609 23181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:39.231817 23172 server_base.cc:1034] running on GCE node
I20260812 01:03:39.232618 23172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:39.233595 23172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:39.234848 23172 hybrid_clock.cc:648] HybridClock initialized: now 1786496619234804 us; error 48 us; skew 500 ppm
I20260812 01:03:39.235206 23172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:03:39.237659 23172 webserver.cc:460] Webserver started at http://127.21.201.66:45525/ using document root <none> and password file <none>
I20260812 01:03:39.238375 23172 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:39.238521 23172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:39.238845 23172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:39.241808 23172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-1/data/instance:
uuid: "69d96acc012244a888b518f3ce60c258"
format_stamp: "Formatted at 2026-08-12 01:03:39 on dist-test-slave-7f01"
server_key: "920e690d59890a8bbfa3381f831d1f28"
server_key_iv: "e73e97c0343983339fed6dd73e7f0202"
server_key_version: "encryptionkey@0"
I20260812 01:03:39.242627 23172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance:
uuid: "69d96acc012244a888b518f3ce60c258"
format_stamp: "Formatted at 2026-08-12 01:03:39 on dist-test-slave-7f01"
server_key: "920e690d59890a8bbfa3381f831d1f28"
server_key_iv: "e73e97c0343983339fed6dd73e7f0202"
server_key_version: "encryptionkey@0"
I20260812 01:03:39.247936 23172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 01:03:39.251426 23195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:39.252841 23172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:03:39.253053 23172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "69d96acc012244a888b518f3ce60c258"
format_stamp: "Formatted at 2026-08-12 01:03:39 on dist-test-slave-7f01"
server_key: "920e690d59890a8bbfa3381f831d1f28"
server_key_iv: "e73e97c0343983339fed6dd73e7f0202"
server_key_version: "encryptionkey@0"
I20260812 01:03:39.253297 23172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:39.269917 23172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:39.270742 23172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:39.271072 23172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:39.272141 23172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:03:39.273700 23172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:03:39.273828 23172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:39.273963 23172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:03:39.274061 23172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:39.309571 23172 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:42355
I20260812 01:03:39.309628 23377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:42355 every 8 connection(s)
I20260812 01:03:39.310963 23172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 01:03:39.311978 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 23172
I20260812 01:03:39.312196 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance
I20260812 01:03:39.312824 22309 external_mini_cluster.cc:1428] Setting key b824432773a320a195891235a9373502
I20260812 01:03:39.316455 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:0
--local_ip_for_outbound_sockets=127.21.201.67
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:46519
--builtin_ntp_servers=127.21.201.84:41979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 01:03:39.326924 23169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:39.314132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:39087 (local address 127.21.201.126:46519)
0812 01:03:39.314338 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:39.314344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:39.316008 (+ 1664us) server_negotiation.cc:407] Connection header received
0812 01:03:39.318917 (+ 2909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:39.318922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:39.318995 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:39.319086 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:39.320528 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:39.321264 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:39.323810 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:39.324390 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:39.324504 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:39.326176 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:39.326181 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:39.326183 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:39.326203 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:39.326238 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:39.326250 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:39.326323 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:39.326590 (+ 267us) server_negotiation.cc:299] Negotiation successful
0812 01:03:39.326674 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 01:03:39.328428 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:39.314638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46519 (local address 127.21.201.66:39087)
0812 01:03:39.315834 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:39.315883 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:39.318607 (+ 2724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:39.319342 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:39.319355 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:39.319485 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:03:39.320320 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:39.320349 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:39.321428 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:39.321444 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:39.323549 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:39.323568 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:39.324639 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:39.324650 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:39.324994 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:39.325792 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:03:39.325798 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:03:39.325855 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:03:39.325966 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:03:39.326352 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:03:39.326364 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:03:39.326375 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 01:03:39.326568 (+ 193us) client_negotiation.cc:240] Negotiation successful
0812 01:03:39.327003 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":380,"threads_started":1}
I20260812 01:03:39.330004 23378 heartbeater.cc:347] Connected to a master server at 127.21.201.126:46519
I20260812 01:03:39.330327 23378 heartbeater.cc:464] Registering TS with master...
I20260812 01:03:39.331076 23378 heartbeater.cc:511] Master 127.21.201.126:46519 requested a full tablet report, sending...
I20260812 01:03:39.332767 22848 ts_manager.cc:194] Registered new tserver with Master: 69d96acc012244a888b518f3ce60c258 (127.21.201.66:42355)
I20260812 01:03:39.334100 22848 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.201.66:39087
W20260812 01:03:39.672027 23386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:39.672338 23386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:39.672434 23386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:39.679822 23386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:39.679919 23386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:39.680030 23386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 01:03:39.689343 23386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:41979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:46519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:39.690653 23386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:39.692459 23386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:39.705964 23392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:39.706055 23396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:39.706000 23394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:39.706938 23386 server_base.cc:1034] running on GCE node
I20260812 01:03:39.707602 23386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:39.708626 23386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:39.709859 23386 hybrid_clock.cc:648] HybridClock initialized: now 1786496619709776 us; error 71 us; skew 500 ppm
I20260812 01:03:39.710182 23386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:03:39.722051 23386 webserver.cc:460] Webserver started at http://127.21.201.67:41521/ using document root <none> and password file <none>
I20260812 01:03:39.722795 23386 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:39.722939 23386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:39.723266 23386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:39.726321 23386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-2/data/instance:
uuid: "423389390ac14fd5be9379280f6e7597"
format_stamp: "Formatted at 2026-08-12 01:03:39 on dist-test-slave-7f01"
server_key: "49afa92e6e3354e65b4cffe3ea848e89"
server_key_iv: "a9de5f37619df793976ec9f78a4354fd"
server_key_version: "encryptionkey@0"
I20260812 01:03:39.727171 23386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance:
uuid: "423389390ac14fd5be9379280f6e7597"
format_stamp: "Formatted at 2026-08-12 01:03:39 on dist-test-slave-7f01"
server_key: "49afa92e6e3354e65b4cffe3ea848e89"
server_key_iv: "a9de5f37619df793976ec9f78a4354fd"
server_key_version: "encryptionkey@0"
I20260812 01:03:39.732705 23386 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260812 01:03:39.736531 23405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:39.737924 23386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:03:39.738067 23386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "423389390ac14fd5be9379280f6e7597"
format_stamp: "Formatted at 2026-08-12 01:03:39 on dist-test-slave-7f01"
server_key: "49afa92e6e3354e65b4cffe3ea848e89"
server_key_iv: "a9de5f37619df793976ec9f78a4354fd"
server_key_version: "encryptionkey@0"
I20260812 01:03:39.738250 23386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:39.760603 23386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:39.761469 23386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:39.761786 23386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:39.762873 23386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:03:39.764473 23386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:03:39.764562 23386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:39.764714 23386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:03:39.764807 23386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:39.800524 23386 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:33631
I20260812 01:03:39.800583 23581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:33631 every 8 connection(s)
I20260812 01:03:39.801817 23386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 01:03:39.810283 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 23386
I20260812 01:03:39.810488 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance
I20260812 01:03:39.810946 22309 external_mini_cluster.cc:1428] Setting key 6385830444197ecc7166d5c9c0aea4a3
I20260812 01:03:39.814543 23169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:39.804422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:45635 (local address 127.21.201.126:46519)
0812 01:03:39.804619 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:39.804624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:39.805997 (+ 1373us) server_negotiation.cc:407] Connection header received
0812 01:03:39.807833 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:39.807838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:39.807908 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:39.808007 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:39.809007 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:39.809750 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:39.811430 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:39.811912 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:39.812076 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:39.813670 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:39.813684 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:39.813687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:39.813704 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:39.813739 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:39.813751 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:39.813856 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:39.814095 (+ 239us) server_negotiation.cc:299] Negotiation successful
0812 01:03:39.814198 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 01:03:39.815505 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:39.804930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:46519 (local address 127.21.201.67:45635)
0812 01:03:39.805829 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:39.805864 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:39.807552 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:39.808199 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:39.808206 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:39.808286 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:03:39.808847 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:39.808863 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:39.809880 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:39.809883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:39.811264 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:39.811278 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:39.812268 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:39.812277 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:39.812536 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:39.813344 (+ 808us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:03:39.813346 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:03:39.813384 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:03:39.813465 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:03:39.813871 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:03:39.813883 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:03:39.813892 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 01:03:39.814079 (+ 187us) client_negotiation.cc:240] Negotiation successful
0812 01:03:39.814469 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":284,"threads_started":1}
I20260812 01:03:39.817010 23583 heartbeater.cc:347] Connected to a master server at 127.21.201.126:46519
I20260812 01:03:39.817333 23583 heartbeater.cc:464] Registering TS with master...
I20260812 01:03:39.818094 23583 heartbeater.cc:511] Master 127.21.201.126:46519 requested a full tablet report, sending...
I20260812 01:03:39.819527 22848 ts_manager.cc:194] Registered new tserver with Master: 423389390ac14fd5be9379280f6e7597 (127.21.201.67:33631)
I20260812 01:03:39.820453 22848 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.201.67:45635
I20260812 01:03:39.826134 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:03:39.838130 23169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:39.831260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50524 (local address 127.21.201.126:46519)
0812 01:03:39.831901 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:39.831906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:39.831960 (+ 54us) server_negotiation.cc:407] Connection header received
0812 01:03:39.832023 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:39.832027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:39.832103 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:39.832251 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:39.832953 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:39.833794 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:39.835276 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:39.835783 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:39.835934 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:39.837330 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:39.837335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:39.837338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:39.837356 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:39.837391 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:39.837404 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:39.837574 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:39.837783 (+ 209us) server_negotiation.cc:299] Negotiation successful
0812 01:03:39.837845 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260812 01:03:39.842484 22848 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:03:39.849594 22848 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:03:39.862351 23165 heartbeater.cc:503] Master 127.21.201.126:46519 was elected leader, sending a full tablet report...
I20260812 01:03:39.893597 23601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:39.864251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58206 (local address 127.21.201.65:46011)
0812 01:03:39.877538 (+ 13287us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:39.877553 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:39.885032 (+ 7479us) server_negotiation.cc:407] Connection header received
0812 01:03:39.885176 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:39.885185 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:39.885410 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:39.885622 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:39.887251 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:39.888888 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:39.891258 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:39.892300 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:39.892476 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:39.892782 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:39.893001 (+ 219us) server_negotiation.cc:299] Negotiation successful
0812 01:03:39.893174 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13028,"thread_start_us":13287,"threads_started":1}
I20260812 01:03:39.893618 23603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:39.863629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:46011 (local address 127.0.0.1:58206)
0812 01:03:39.884229 (+ 20600us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:39.884256 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:39.884457 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:39.886050 (+ 1593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:39.886056 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:39.886083 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:39.887022 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:39.887038 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:39.889080 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:39.889089 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:39.891044 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:39.891057 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:39.892515 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:39.892521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:39.892710 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:39.892773 (+ 63us) client_negotiation.cc:769] Sending connection context
0812 01:03:39.892956 (+ 183us) client_negotiation.cc:240] Negotiation successful
0812 01:03:39.893174 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20400,"mutex_wait_us":65,"thread_start_us":21133,"threads_started":1}
I20260812 01:03:39.894073 23598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:39.863227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:33631 (local address 127.0.0.1:58550)
0812 01:03:39.865077 (+ 1850us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:39.865139 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:39.865343 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:39.885620 (+ 20277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:39.885628 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:39.885662 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:39.886722 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:39.886740 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:39.888952 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:39.888958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:39.891259 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:39.891274 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:39.893206 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:39.893212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:39.893415 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:39.893450 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 01:03:39.893522 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 01:03:39.893667 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1526,"mutex_wait_us":89,"thread_start_us":1333,"threads_started":1}
I20260812 01:03:39.895408 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:39.865415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:42355 (local address 127.0.0.1:45352)
0812 01:03:39.885031 (+ 19616us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:39.885052 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:39.885163 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:39.886311 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:39.886316 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:39.886343 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:39.888191 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:39.888206 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:39.890111 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:39.890117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:39.892359 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:39.892374 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:39.894599 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:39.894605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:39.894789 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:39.894819 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 01:03:39.894906 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 01:03:39.895034 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19492,"mutex_wait_us":1,"thread_start_us":18477,"threads_started":1}
I20260812 01:03:39.895561 23604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:39.866023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45352 (local address 127.21.201.66:42355)
0812 01:03:39.882306 (+ 16283us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:39.882315 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:39.885285 (+ 2970us) server_negotiation.cc:407] Connection header received
0812 01:03:39.885417 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:39.885425 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:39.885642 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:39.885832 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:39.888429 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:39.889948 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:39.892621 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:39.894077 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:39.894216 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:39.894468 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:39.895000 (+ 532us) server_negotiation.cc:299] Negotiation successful
0812 01:03:39.895183 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15840,"thread_start_us":15883,"threads_started":1}
I20260812 01:03:39.896564 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:39.863529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58550 (local address 127.21.201.67:33631)
0812 01:03:39.884125 (+ 20596us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:39.884133 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:39.884175 (+ 42us) server_negotiation.cc:407] Connection header received
0812 01:03:39.884308 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:39.884316 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:39.884599 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:39.884837 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:39.887132 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:39.888796 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:39.891480 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:39.892980 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:39.894344 (+ 1364us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:39.894602 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:39.894671 (+ 69us) server_negotiation.cc:299] Negotiation successful
0812 01:03:39.896172 (+ 1501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20394,"thread_start_us":21578,"threads_started":1}
I20260812 01:03:39.898104 23045 tablet_service.cc:1467] Processing CreateTablet for tablet d697149e57da4fadb2af845634018143 (DEFAULT_TABLE table=test-table [id=4ad9f95786174f9eb0e6b39c442cb436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:03:39.898744 23266 tablet_service.cc:1467] Processing CreateTablet for tablet d697149e57da4fadb2af845634018143 (DEFAULT_TABLE table=test-table [id=4ad9f95786174f9eb0e6b39c442cb436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:03:39.900015 23045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d697149e57da4fadb2af845634018143. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:03:39.900065 23266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d697149e57da4fadb2af845634018143. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:03:39.900033 23481 tablet_service.cc:1467] Processing CreateTablet for tablet d697149e57da4fadb2af845634018143 (DEFAULT_TABLE table=test-table [id=4ad9f95786174f9eb0e6b39c442cb436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:03:39.901382 23481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d697149e57da4fadb2af845634018143. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:03:39.912191 23606 tablet_bootstrap.cc:492] T d697149e57da4fadb2af845634018143 P 69d96acc012244a888b518f3ce60c258: Bootstrap starting.
I20260812 01:03:39.916329 23606 tablet_bootstrap.cc:654] T d697149e57da4fadb2af845634018143 P 69d96acc012244a888b518f3ce60c258: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:39.917354 23606 log.cc:824] T d697149e57da4fadb2af845634018143 P 69d96acc012244a888b518f3ce60c258: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:39.918802 23611 tablet_bootstrap.cc:492] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597: Bootstrap starting.
I20260812 01:03:39.920049 23610 tablet_bootstrap.cc:492] T d697149e57da4fadb2af845634018143 P fc9d3bf3f0804bf1bc7b274416f7ebad: Bootstrap starting.
I20260812 01:03:39.920673 23606 tablet_bootstrap.cc:492] T d697149e57da4fadb2af845634018143 P 69d96acc012244a888b518f3ce60c258: No bootstrap required, opened a new log
I20260812 01:03:39.920943 23606 ts_tablet_manager.cc:1397] T d697149e57da4fadb2af845634018143 P 69d96acc012244a888b518f3ce60c258: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260812 01:03:39.923645 23610 tablet_bootstrap.cc:654] T d697149e57da4fadb2af845634018143 P fc9d3bf3f0804bf1bc7b274416f7ebad: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:39.924880 23610 log.cc:824] T d697149e57da4fadb2af845634018143 P fc9d3bf3f0804bf1bc7b274416f7ebad: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:39.926697 23611 tablet_bootstrap.cc:654] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:39.927865 23611 log.cc:824] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:39.928087 23610 tablet_bootstrap.cc:492] T d697149e57da4fadb2af845634018143 P fc9d3bf3f0804bf1bc7b274416f7ebad: No bootstrap required, opened a new log
I20260812 01:03:39.928411 23610 ts_tablet_manager.cc:1397] T d697149e57da4fadb2af845634018143 P fc9d3bf3f0804bf1bc7b274416f7ebad: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260812 01:03:39.931609 23606 raft_consensus.cc:348] T d697149e57da4fadb2af845634018143 P 69d96acc012244a888b518f3ce60c258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "423389390ac14fd5be9379280f6e7597" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33631 } } peers { permanent_uuid: "fc9d3bf3f0804bf1bc7b274416f7ebad" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 46011 } } peers { permanent_uuid: "69d96acc012244a888b518f3ce60c258" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42355 } }
I20260812 01:03:39.932211 23606 raft_consensus.cc:374] T d697149e57da4fadb2af845634018143 P 69d96acc012244a888b518f3ce60c258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:39.932343 23606 raft_consensus.cc:729] T d697149e57da4fadb2af845634018143 P 69d96acc012244a888b518f3ce60c258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69d96acc012244a888b518f3ce60c258, State: Initialized, Role: FOLLOWER
I20260812 01:03:39.933022 23606 consensus_queue.cc:260] T d697149e57da4fadb2af845634018143 P 69d96acc012244a888b518f3ce60c258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "423389390ac14fd5be9379280f6e7597" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33631 } } peers { permanent_uuid: "fc9d3bf3f0804bf1bc7b274416f7ebad" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 46011 } } peers { permanent_uuid: "69d96acc012244a888b518f3ce60c258" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42355 } }
I20260812 01:03:39.934295 23378 heartbeater.cc:503] Master 127.21.201.126:46519 was elected leader, sending a full tablet report...
I20260812 01:03:39.934517 23611 tablet_bootstrap.cc:492] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597: No bootstrap required, opened a new log
I20260812 01:03:39.934782 23611 ts_tablet_manager.cc:1397] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.005s
I20260812 01:03:39.934881 23606 ts_tablet_manager.cc:1428] T d697149e57da4fadb2af845634018143 P 69d96acc012244a888b518f3ce60c258: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260812 01:03:39.938704 23610 raft_consensus.cc:348] T d697149e57da4fadb2af845634018143 P fc9d3bf3f0804bf1bc7b274416f7ebad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "423389390ac14fd5be9379280f6e7597" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33631 } } peers { permanent_uuid: "fc9d3bf3f0804bf1bc7b274416f7ebad" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 46011 } } peers { permanent_uuid: "69d96acc012244a888b518f3ce60c258" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42355 } }
I20260812 01:03:39.939177 23610 raft_consensus.cc:374] T d697149e57da4fadb2af845634018143 P fc9d3bf3f0804bf1bc7b274416f7ebad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:39.939301 23610 raft_consensus.cc:729] T d697149e57da4fadb2af845634018143 P fc9d3bf3f0804bf1bc7b274416f7ebad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc9d3bf3f0804bf1bc7b274416f7ebad, State: Initialized, Role: FOLLOWER
I20260812 01:03:39.939970 23610 consensus_queue.cc:260] T d697149e57da4fadb2af845634018143 P fc9d3bf3f0804bf1bc7b274416f7ebad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "423389390ac14fd5be9379280f6e7597" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33631 } } peers { permanent_uuid: "fc9d3bf3f0804bf1bc7b274416f7ebad" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 46011 } } peers { permanent_uuid: "69d96acc012244a888b518f3ce60c258" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42355 } }
I20260812 01:03:39.941882 23610 ts_tablet_manager.cc:1428] T d697149e57da4fadb2af845634018143 P fc9d3bf3f0804bf1bc7b274416f7ebad: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260812 01:03:39.943636 23611 raft_consensus.cc:348] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "423389390ac14fd5be9379280f6e7597" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33631 } } peers { permanent_uuid: "fc9d3bf3f0804bf1bc7b274416f7ebad" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 46011 } } peers { permanent_uuid: "69d96acc012244a888b518f3ce60c258" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42355 } }
I20260812 01:03:39.944082 23611 raft_consensus.cc:374] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:39.944238 23611 raft_consensus.cc:729] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 423389390ac14fd5be9379280f6e7597, State: Initialized, Role: FOLLOWER
I20260812 01:03:39.944859 23611 consensus_queue.cc:260] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "423389390ac14fd5be9379280f6e7597" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33631 } } peers { permanent_uuid: "fc9d3bf3f0804bf1bc7b274416f7ebad" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 46011 } } peers { permanent_uuid: "69d96acc012244a888b518f3ce60c258" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42355 } }
I20260812 01:03:39.945999 23583 heartbeater.cc:503] Master 127.21.201.126:46519 was elected leader, sending a full tablet report...
I20260812 01:03:39.946517 23611 ts_tablet_manager.cc:1428] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260812 01:03:40.053604 23584 tablet.cc:2369] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:03:40.063212 23379 tablet.cc:2369] T d697149e57da4fadb2af845634018143 P 69d96acc012244a888b518f3ce60c258: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:03:40.090081 23167 tablet.cc:2369] T d697149e57da4fadb2af845634018143 P fc9d3bf3f0804bf1bc7b274416f7ebad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:03:40.090677 23624 raft_consensus.cc:482] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:03:40.091041 23624 raft_consensus.cc:504] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "423389390ac14fd5be9379280f6e7597" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33631 } } peers { permanent_uuid: "fc9d3bf3f0804bf1bc7b274416f7ebad" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 46011 } } peers { permanent_uuid: "69d96acc012244a888b518f3ce60c258" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42355 } }
I20260812 01:03:40.092823 23624 leader_election.cc:288] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc9d3bf3f0804bf1bc7b274416f7ebad (127.21.201.65:46011), 69d96acc012244a888b518f3ce60c258 (127.21.201.66:42355)
I20260812 01:03:40.099754 23601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:40.093379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:49047 (local address 127.21.201.65:46011)
0812 01:03:40.093600 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:40.093607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:40.093790 (+ 183us) server_negotiation.cc:407] Connection header received
0812 01:03:40.094031 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:40.094036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:40.094133 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:40.094258 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:40.094830 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:40.095813 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:40.097728 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:40.098735 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:40.098878 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:40.098929 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:40.099301 (+ 372us) server_negotiation.cc:299] Negotiation successful
0812 01:03:40.099461 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 01:03:40.099864 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:40.093206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:46011 (local address 127.21.201.67:49047)
0812 01:03:40.093653 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:40.093682 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:40.093860 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:40.094414 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:40.094419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:40.094455 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:40.094660 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:40.094672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:40.095965 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:40.095970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:40.097562 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:40.097575 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:40.098899 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:40.098905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:40.099075 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:40.099118 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 01:03:40.099353 (+ 235us) client_negotiation.cc:240] Negotiation successful
0812 01:03:40.099556 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290}
I20260812 01:03:40.100420 23604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:40.093391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:56837 (local address 127.21.201.66:42355)
0812 01:03:40.093600 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:40.093606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:40.094265 (+ 659us) server_negotiation.cc:407] Connection header received
0812 01:03:40.094446 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:40.094451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:40.094543 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:40.094651 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:40.095359 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:40.096339 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:40.098259 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:40.099196 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:40.099339 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:40.099406 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:40.099977 (+ 571us) server_negotiation.cc:299] Negotiation successful
0812 01:03:40.100135 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 01:03:40.100878 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:40.093255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:42355 (local address 127.21.201.67:56837)
0812 01:03:40.094146 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:40.094164 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:40.094294 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:40.094674 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:40.094677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:40.094701 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:40.095149 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:40.095162 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:40.096516 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:40.096520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:40.098061 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:40.098076 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:40.099540 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:40.099547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:40.099758 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:40.099800 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 01:03:40.100504 (+ 704us) client_negotiation.cc:240] Negotiation successful
0812 01:03:40.100646 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":535,"threads_started":1}
I20260812 01:03:40.101035 23082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d697149e57da4fadb2af845634018143" candidate_uuid: "423389390ac14fd5be9379280f6e7597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc9d3bf3f0804bf1bc7b274416f7ebad" is_pre_election: true
I20260812 01:03:40.101691 23082 raft_consensus.cc:2454] T d697149e57da4fadb2af845634018143 P fc9d3bf3f0804bf1bc7b274416f7ebad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 423389390ac14fd5be9379280f6e7597 in term 0.
I20260812 01:03:40.101962 23304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d697149e57da4fadb2af845634018143" candidate_uuid: "423389390ac14fd5be9379280f6e7597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69d96acc012244a888b518f3ce60c258" is_pre_election: true
I20260812 01:03:40.102499 23407 leader_election.cc:302] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 423389390ac14fd5be9379280f6e7597, fc9d3bf3f0804bf1bc7b274416f7ebad; no voters:
I20260812 01:03:40.102568 23304 raft_consensus.cc:2454] T d697149e57da4fadb2af845634018143 P 69d96acc012244a888b518f3ce60c258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 423389390ac14fd5be9379280f6e7597 in term 0.
I20260812 01:03:40.102977 23624 raft_consensus.cc:2781] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:03:40.103144 23624 raft_consensus.cc:482] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:03:40.103257 23624 raft_consensus.cc:3037] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:40.105882 23624 raft_consensus.cc:504] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "423389390ac14fd5be9379280f6e7597" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33631 } } peers { permanent_uuid: "fc9d3bf3f0804bf1bc7b274416f7ebad" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 46011 } } peers { permanent_uuid: "69d96acc012244a888b518f3ce60c258" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42355 } }
I20260812 01:03:40.106504 23624 leader_election.cc:288] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597 [CANDIDATE]: Term 1 election: Requested vote from peers fc9d3bf3f0804bf1bc7b274416f7ebad (127.21.201.65:46011), 69d96acc012244a888b518f3ce60c258 (127.21.201.66:42355)
I20260812 01:03:40.106933 23082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d697149e57da4fadb2af845634018143" candidate_uuid: "423389390ac14fd5be9379280f6e7597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc9d3bf3f0804bf1bc7b274416f7ebad"
I20260812 01:03:40.106972 23304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d697149e57da4fadb2af845634018143" candidate_uuid: "423389390ac14fd5be9379280f6e7597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69d96acc012244a888b518f3ce60c258"
I20260812 01:03:40.107169 23082 raft_consensus.cc:3037] T d697149e57da4fadb2af845634018143 P fc9d3bf3f0804bf1bc7b274416f7ebad [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:40.107190 23304 raft_consensus.cc:3037] T d697149e57da4fadb2af845634018143 P 69d96acc012244a888b518f3ce60c258 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:40.109565 23304 raft_consensus.cc:2454] T d697149e57da4fadb2af845634018143 P 69d96acc012244a888b518f3ce60c258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 423389390ac14fd5be9379280f6e7597 in term 1.
I20260812 01:03:40.109571 23082 raft_consensus.cc:2454] T d697149e57da4fadb2af845634018143 P fc9d3bf3f0804bf1bc7b274416f7ebad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 423389390ac14fd5be9379280f6e7597 in term 1.
I20260812 01:03:40.110008 23407 leader_election.cc:302] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 423389390ac14fd5be9379280f6e7597, fc9d3bf3f0804bf1bc7b274416f7ebad; no voters:
I20260812 01:03:40.110311 23624 raft_consensus.cc:2781] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:03:40.110782 23624 raft_consensus.cc:686] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597 [term 1 LEADER]: Becoming Leader. State: Replica: 423389390ac14fd5be9379280f6e7597, State: Running, Role: LEADER
I20260812 01:03:40.111295 23624 consensus_queue.cc:237] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "423389390ac14fd5be9379280f6e7597" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33631 } } peers { permanent_uuid: "fc9d3bf3f0804bf1bc7b274416f7ebad" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 46011 } } peers { permanent_uuid: "69d96acc012244a888b518f3ce60c258" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42355 } }
I20260812 01:03:40.116382 22845 catalog_manager.cc:5495] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597 reported cstate change: term changed from 0 to 1, leader changed from <none> to 423389390ac14fd5be9379280f6e7597 (127.21.201.67). New cstate: current_term: 1 leader_uuid: "423389390ac14fd5be9379280f6e7597" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "423389390ac14fd5be9379280f6e7597" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 33631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc9d3bf3f0804bf1bc7b274416f7ebad" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 46011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69d96acc012244a888b518f3ce60c258" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42355 } health_report { overall_health: UNKNOWN } } }
I20260812 01:03:40.173650 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:40.167816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58564 (local address 127.21.201.67:33631)
0812 01:03:40.168045 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:40.168053 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:40.168078 (+ 25us) server_negotiation.cc:407] Connection header received
0812 01:03:40.168333 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:40.168337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:40.168432 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:40.168608 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:03:40.169035 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:40.169778 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:40.171207 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:40.171656 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:40.171834 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:40.172468 (+ 634us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:03:40.172992 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:03:40.173188 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:40.173355 (+ 167us) server_negotiation.cc:299] Negotiation successful
0812 01:03:40.173401 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 01:03:40.182534 23304 raft_consensus.cc:1261] T d697149e57da4fadb2af845634018143 P 69d96acc012244a888b518f3ce60c258 [term 1 FOLLOWER]: Refusing update from remote peer 423389390ac14fd5be9379280f6e7597: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:03:40.182576 23082 raft_consensus.cc:1261] T d697149e57da4fadb2af845634018143 P fc9d3bf3f0804bf1bc7b274416f7ebad [term 1 FOLLOWER]: Refusing update from remote peer 423389390ac14fd5be9379280f6e7597: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:03:40.183566 23627 consensus_queue.cc:1035] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69d96acc012244a888b518f3ce60c258" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:03:40.184031 23624 consensus_queue.cc:1035] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc9d3bf3f0804bf1bc7b274416f7ebad" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 46011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:03:40.197108 23633 mvcc.cc:205] Tried to move back new op lower bound from 7317490156255383552 to 7317490155979812864. Current Snapshot: MvccSnapshot[applied={T|T < 7317490156255383552}]
I20260812 01:03:40.215025 22848 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:03:40.215365 22848 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:03:40.219189 22848 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 905879d6449448a8b3b1a8327bc37095: Sending DeleteTablet for 3 replicas of tablet d697149e57da4fadb2af845634018143
I20260812 01:03:40.220391 23481 tablet_service.cc:1514] Processing DeleteTablet for tablet d697149e57da4fadb2af845634018143 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:03:40 UTC) from {username='slave'} at 127.0.0.1:58550
I20260812 01:03:40.220875 23045 tablet_service.cc:1514] Processing DeleteTablet for tablet d697149e57da4fadb2af845634018143 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:03:40 UTC) from {username='slave'} at 127.0.0.1:58206
I20260812 01:03:40.221483 23266 tablet_service.cc:1514] Processing DeleteTablet for tablet d697149e57da4fadb2af845634018143 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:03:40 UTC) from {username='slave'} at 127.0.0.1:45352
I20260812 01:03:40.222394 23659 tablet_replica.cc:330] T d697149e57da4fadb2af845634018143 P 69d96acc012244a888b518f3ce60c258: stopping tablet replica
I20260812 01:03:40.222995 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 22939
I20260812 01:03:40.228302 23659 raft_consensus.cc:2229] T d697149e57da4fadb2af845634018143 P 69d96acc012244a888b518f3ce60c258 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:03:40.229110 23659 raft_consensus.cc:2258] T d697149e57da4fadb2af845634018143 P 69d96acc012244a888b518f3ce60c258 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:03:40.232033 23659 ts_tablet_manager.cc:1905] T d697149e57da4fadb2af845634018143 P 69d96acc012244a888b518f3ce60c258: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:03:40.236377 23655 tablet_replica.cc:330] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597: stopping tablet replica
I20260812 01:03:40.243471 23659 ts_tablet_manager.cc:1918] T d697149e57da4fadb2af845634018143 P 69d96acc012244a888b518f3ce60c258: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:03:40.246140 23655 raft_consensus.cc:2229] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597 [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:03:40.246829 23659 log.cc:1196] T d697149e57da4fadb2af845634018143 P 69d96acc012244a888b518f3ce60c258: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/d697149e57da4fadb2af845634018143
I20260812 01:03:40.246986 23658 tablet_replica.cc:330] T d697149e57da4fadb2af845634018143 P fc9d3bf3f0804bf1bc7b274416f7ebad: stopping tablet replica
I20260812 01:03:40.247459 23658 raft_consensus.cc:2229] T d697149e57da4fadb2af845634018143 P fc9d3bf3f0804bf1bc7b274416f7ebad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:03:40.248119 23658 raft_consensus.cc:2258] T d697149e57da4fadb2af845634018143 P fc9d3bf3f0804bf1bc7b274416f7ebad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:03:40.248303 23659 ts_tablet_manager.cc:1939] T d697149e57da4fadb2af845634018143 P 69d96acc012244a888b518f3ce60c258: Deleting consensus metadata
I20260812 01:03:40.251381 22827 catalog_manager.cc:4841] TS 69d96acc012244a888b518f3ce60c258 (127.21.201.66:42355): tablet d697149e57da4fadb2af845634018143 (table test-table [id=4ad9f95786174f9eb0e6b39c442cb436]) successfully deleted
I20260812 01:03:40.256011 23655 raft_consensus.cc:2258] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:03:40.259670 23658 ts_tablet_manager.cc:1905] T d697149e57da4fadb2af845634018143 P fc9d3bf3f0804bf1bc7b274416f7ebad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:03:40.259676 23655 ts_tablet_manager.cc:1905] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:03:40.267333 23658 ts_tablet_manager.cc:1918] T d697149e57da4fadb2af845634018143 P fc9d3bf3f0804bf1bc7b274416f7ebad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:03:40.267522 23658 log.cc:1196] T d697149e57da4fadb2af845634018143 P fc9d3bf3f0804bf1bc7b274416f7ebad: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-0/wal/wals/d697149e57da4fadb2af845634018143
I20260812 01:03:40.268635 23655 ts_tablet_manager.cc:1918] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:03:40.268884 23655 log.cc:1196] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/d697149e57da4fadb2af845634018143
I20260812 01:03:40.269608 23655 ts_tablet_manager.cc:1939] T d697149e57da4fadb2af845634018143 P 423389390ac14fd5be9379280f6e7597: Deleting consensus metadata
I20260812 01:03:40.271884 22827 catalog_manager.cc:4841] TS 423389390ac14fd5be9379280f6e7597 (127.21.201.67:33631): tablet d697149e57da4fadb2af845634018143 (table test-table [id=4ad9f95786174f9eb0e6b39c442cb436]) successfully deleted
I20260812 01:03:40.279364 23658 ts_tablet_manager.cc:1939] T d697149e57da4fadb2af845634018143 P fc9d3bf3f0804bf1bc7b274416f7ebad: Deleting consensus metadata
I20260812 01:03:40.285068 22820 catalog_manager.cc:4841] TS fc9d3bf3f0804bf1bc7b274416f7ebad (127.21.201.65:46011): tablet d697149e57da4fadb2af845634018143 (table test-table [id=4ad9f95786174f9eb0e6b39c442cb436]) successfully deleted
I20260812 01:03:40.288025 23601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:40.224290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58220 (local address 127.21.201.65:46011)
0812 01:03:40.224479 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:40.224488 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:40.249130 (+ 24642us) server_negotiation.cc:407] Connection header received
0812 01:03:40.249238 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:40.249243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:40.249555 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:40.249697 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:40.250414 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:40.251709 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:40.254351 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:40.255260 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:40.255381 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:40.257503 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:40.257515 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:40.257533 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:40.286530 (+ 28997us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:40.286621 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:40.286667 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:40.286852 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:40.287243 (+ 391us) server_negotiation.cc:299] Negotiation successful
0812 01:03:40.287510 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 01:03:40.290436 23156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:40.411722 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 22939
I20260812 01:03:40.438536 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 23172
I20260812 01:03:40.447525 23604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:40.439726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45360 (local address 127.21.201.66:42355)
0812 01:03:40.439970 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:40.439977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:40.440000 (+ 23us) server_negotiation.cc:407] Connection header received
0812 01:03:40.440076 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:40.440083 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:40.440196 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:40.440316 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:40.440727 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:40.441895 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:40.443428 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:40.444245 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:40.444382 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:40.445482 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:40.445493 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:40.445513 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:40.446343 (+ 830us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:40.446421 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:40.446447 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:40.446563 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:40.446700 (+ 137us) server_negotiation.cc:299] Negotiation successful
0812 01:03:40.446829 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 01:03:40.448303 23370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:40.541693 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 23172
I20260812 01:03:40.574050 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 23386
I20260812 01:03:40.583390 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:40.575004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58578 (local address 127.21.201.67:33631)
0812 01:03:40.575210 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:40.575216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:40.575245 (+ 29us) server_negotiation.cc:407] Connection header received
0812 01:03:40.575358 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:40.575364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:40.575504 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:40.575653 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:40.576191 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:40.577439 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:40.578849 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:40.579538 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:40.579696 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:40.581023 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:40.581040 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:40.581063 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:40.582046 (+ 983us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:40.582119 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:40.582151 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:40.582271 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:40.582400 (+ 129us) server_negotiation.cc:299] Negotiation successful
0812 01:03:40.582512 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 01:03:40.584278 23577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:40.714529 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 23386
I20260812 01:03:40.742077 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 22793
I20260812 01:03:40.743371 22895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:40.859637 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 22793
2026-08-12T01:03:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3195 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:03:40 dist-test-slave-7f01 krb5kdc[23673](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:03:40 dist-test-slave-7f01 krb5kdc[23673](info): set up 2 sockets
Aug 12 01:03:40 dist-test-slave-7f01 krb5kdc[23673](info): commencing operation
krb5kdc: starting...
W20260812 01:03:42.288376 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.376s user 0.004s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:03:42 dist-test-slave-7f01 krb5kdc[23673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:03:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:03:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 01:03:42.609378 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:44891
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:44391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:44891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:03:42.960724 23696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:42.961099 23696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:42.961273 23696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:42.968526 23696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:03:42.968677 23696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:42.968791 23696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:42.968907 23696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:03:42.968997 23696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:03:42.978797 23696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:44391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:44891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:44891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:42.980172 23696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:42.981469 23696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:42.993486 23702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:42.993418 23705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:42.993448 23703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:42.994868 23696 server_base.cc:1034] running on GCE node
I20260812 01:03:42.995610 23696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:42.997020 23696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:42.998265 23696 hybrid_clock.cc:648] HybridClock initialized: now 1786496622998222 us; error 43 us; skew 500 ppm
I20260812 01:03:42.998615 23696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:03:42 dist-test-slave-7f01 krb5kdc[23673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496622, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:03:43.001755 23696 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 01:03:43.003226 23696 webserver.cc:460] Webserver started at http://127.21.201.126:32861/ using document root <none> and password file <none>
I20260812 01:03:43.004007 23696 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:43.004160 23696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:43.004474 23696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:43.007791 23696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "a404298d7e5c4179994243309f373faf"
format_stamp: "Formatted at 2026-08-12 01:03:43 on dist-test-slave-7f01"
server_key: "7a5e2f27906ac0acbdccd081008fda6b"
server_key_iv: "6bc4cf8c0dd1afec5990852d6a9793d0"
server_key_version: "encryptionkey@0"
I20260812 01:03:43.008610 23696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "a404298d7e5c4179994243309f373faf"
format_stamp: "Formatted at 2026-08-12 01:03:43 on dist-test-slave-7f01"
server_key: "7a5e2f27906ac0acbdccd081008fda6b"
server_key_iv: "6bc4cf8c0dd1afec5990852d6a9793d0"
server_key_version: "encryptionkey@0"
I20260812 01:03:43.014075 23696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:03:43.017634 23720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:43.018966 23696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 01:03:43.019183 23696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "a404298d7e5c4179994243309f373faf"
format_stamp: "Formatted at 2026-08-12 01:03:43 on dist-test-slave-7f01"
server_key: "7a5e2f27906ac0acbdccd081008fda6b"
server_key_iv: "6bc4cf8c0dd1afec5990852d6a9793d0"
server_key_version: "encryptionkey@0"
I20260812 01:03:43.019399 23696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:43.037293 23696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:43.041850 23696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:43.042150 23696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:43.061776 23696 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:44891
I20260812 01:03:43.061806 23814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:44891 every 8 connection(s)
I20260812 01:03:43.063212 23696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:03:43.066974 23815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:03:43.071048 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 23696
I20260812 01:03:43.071249 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:03:43.071725 22309 external_mini_cluster.cc:1428] Setting key 5074050dba40ea8697e6faab2aa5f041
I20260812 01:03:43.078395 23815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a404298d7e5c4179994243309f373faf: Bootstrap starting.
I20260812 01:03:43.082158 23815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a404298d7e5c4179994243309f373faf: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:43.083353 23815 log.cc:824] T 00000000000000000000000000000000 P a404298d7e5c4179994243309f373faf: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:43.086833 23815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a404298d7e5c4179994243309f373faf: No bootstrap required, opened a new log
Aug 12 01:03:43 dist-test-slave-7f01 krb5kdc[23673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:03:43.095536 23815 raft_consensus.cc:348] T 00000000000000000000000000000000 P a404298d7e5c4179994243309f373faf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a404298d7e5c4179994243309f373faf" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44891 } }
I20260812 01:03:43.096055 23815 raft_consensus.cc:374] T 00000000000000000000000000000000 P a404298d7e5c4179994243309f373faf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:43.096197 23815 raft_consensus.cc:729] T 00000000000000000000000000000000 P a404298d7e5c4179994243309f373faf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a404298d7e5c4179994243309f373faf, State: Initialized, Role: FOLLOWER
I20260812 01:03:43.096750 23818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:43.073632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50608 (local address 127.21.201.126:44891)
0812 01:03:43.074471 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:43.074483 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:43.074523 (+ 40us) server_negotiation.cc:407] Connection header received
0812 01:03:43.076967 (+ 2444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:43.076988 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:43.077351 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:43.077688 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:43.079607 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:43.081022 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:43.082878 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:43.083601 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:43.083723 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:43.088943 (+ 5220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:43.088953 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:43.088967 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:43.089008 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:43.092472 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:43.093188 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:43.093201 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:43.093206 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:43.093377 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:43.093784 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:43.093790 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:43.093793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:43.094338 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:43.094658 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:43.095081 (+ 423us) server_negotiation.cc:299] Negotiation successful
0812 01:03:43.095495 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":394,"threads_started":1}
I20260812 01:03:43.096830 23815 consensus_queue.cc:260] T 00000000000000000000000000000000 P a404298d7e5c4179994243309f373faf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a404298d7e5c4179994243309f373faf" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44891 } }
I20260812 01:03:43.097188 23815 raft_consensus.cc:388] T 00000000000000000000000000000000 P a404298d7e5c4179994243309f373faf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:03:43.097394 23815 raft_consensus.cc:482] T 00000000000000000000000000000000 P a404298d7e5c4179994243309f373faf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:03:43.097605 23815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a404298d7e5c4179994243309f373faf [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:43.100368 23815 raft_consensus.cc:504] T 00000000000000000000000000000000 P a404298d7e5c4179994243309f373faf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a404298d7e5c4179994243309f373faf" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44891 } }
I20260812 01:03:43.100889 23815 leader_election.cc:302] T 00000000000000000000000000000000 P a404298d7e5c4179994243309f373faf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a404298d7e5c4179994243309f373faf; no voters:
I20260812 01:03:43.101487 23815 leader_election.cc:288] T 00000000000000000000000000000000 P a404298d7e5c4179994243309f373faf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:03:43.101636 23823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a404298d7e5c4179994243309f373faf [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:03:43.102357 23823 raft_consensus.cc:686] T 00000000000000000000000000000000 P a404298d7e5c4179994243309f373faf [term 1 LEADER]: Becoming Leader. State: Replica: a404298d7e5c4179994243309f373faf, State: Running, Role: LEADER
I20260812 01:03:43.102943 23823 consensus_queue.cc:237] T 00000000000000000000000000000000 P a404298d7e5c4179994243309f373faf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a404298d7e5c4179994243309f373faf" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44891 } }
I20260812 01:03:43.103835 23815 sys_catalog.cc:564] T 00000000000000000000000000000000 P a404298d7e5c4179994243309f373faf [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:03:43.105926 23825 sys_catalog.cc:455] T 00000000000000000000000000000000 P a404298d7e5c4179994243309f373faf [sys.catalog]: SysCatalogTable state changed. Reason: New leader a404298d7e5c4179994243309f373faf. Latest consensus state: current_term: 1 leader_uuid: "a404298d7e5c4179994243309f373faf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a404298d7e5c4179994243309f373faf" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44891 } } }
I20260812 01:03:43.105957 23824 sys_catalog.cc:455] T 00000000000000000000000000000000 P a404298d7e5c4179994243309f373faf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a404298d7e5c4179994243309f373faf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a404298d7e5c4179994243309f373faf" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 44891 } } }
I20260812 01:03:43.106403 23825 sys_catalog.cc:458] T 00000000000000000000000000000000 P a404298d7e5c4179994243309f373faf [sys.catalog]: This master's current role is: LEADER
I20260812 01:03:43.106410 23824 sys_catalog.cc:458] T 00000000000000000000000000000000 P a404298d7e5c4179994243309f373faf [sys.catalog]: This master's current role is: LEADER
I20260812 01:03:43.107285 23829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:03:43.113381 23829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:03:43.122709 23829 catalog_manager.cc:1347] Generated new cluster ID: 0b08f8502b2c4579af4ef336153276ab
I20260812 01:03:43.122815 23829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:03:43.138768 23829 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:03:43.140146 23829 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:03:43.152416 23829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a404298d7e5c4179994243309f373faf: Generated new TSK 0
I20260812 01:03:43.153173 23829 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.201.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
Entry for principal kudu/127.21.201.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab.
I20260812 01:03:43.227977 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:0
--local_ip_for_outbound_sockets=127.21.201.65
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:44891
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:44391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:03:43.578716 23855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:43.579063 23855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:43.579241 23855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:43.586457 23855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:43.586606 23855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:43.586804 23855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 01:03:43.596688 23855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:44391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.65.keytab
--principal=kudu/127.21.201.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:44891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:43.598021 23855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:43.599475 23855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:43.613427 23863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:43.613531 23862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:43.613407 23868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:43.614657 23855 server_base.cc:1034] running on GCE node
I20260812 01:03:43.615267 23855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:43.616271 23855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:43.617516 23855 hybrid_clock.cc:648] HybridClock initialized: now 1786496623617463 us; error 57 us; skew 500 ppm
I20260812 01:03:43.617879 23855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:03:43 dist-test-slave-7f01 krb5kdc[23673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496623, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:03:43.621223 23855 init.cc:377] Logged in from keytab as kudu/127.21.201.65@KRBTEST.COM (short username kudu)
I20260812 01:03:43.622825 23855 webserver.cc:460] Webserver started at http://127.21.201.65:33647/ using document root <none> and password file <none>
I20260812 01:03:43.623646 23855 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:43.623822 23855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:43.624145 23855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:43.627130 23855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-0/data/instance:
uuid: "2b6d02dc8fd74f8e89e3557ae36cb23a"
format_stamp: "Formatted at 2026-08-12 01:03:43 on dist-test-slave-7f01"
server_key: "302cfcaf405173a3d98f71d53aabad52"
server_key_iv: "04134afbdf850d3d4dd40c1b26c1e20f"
server_key_version: "encryptionkey@0"
I20260812 01:03:43.627988 23855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance:
uuid: "2b6d02dc8fd74f8e89e3557ae36cb23a"
format_stamp: "Formatted at 2026-08-12 01:03:43 on dist-test-slave-7f01"
server_key: "302cfcaf405173a3d98f71d53aabad52"
server_key_iv: "04134afbdf850d3d4dd40c1b26c1e20f"
server_key_version: "encryptionkey@0"
I20260812 01:03:43.633242 23855 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 01:03:43.636623 23878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:43.638301 23855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:03:43.638492 23855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "2b6d02dc8fd74f8e89e3557ae36cb23a"
format_stamp: "Formatted at 2026-08-12 01:03:43 on dist-test-slave-7f01"
server_key: "302cfcaf405173a3d98f71d53aabad52"
server_key_iv: "04134afbdf850d3d4dd40c1b26c1e20f"
server_key_version: "encryptionkey@0"
I20260812 01:03:43.638706 23855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:43.671424 23855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:43.675585 23855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:43.675910 23855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:43.676903 23855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:03:43.678282 23855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:03:43.678359 23855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:43.678452 23855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:03:43.678498 23855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:43.714020 23855 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:43281
I20260812 01:03:43.714073 24060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:43281 every 8 connection(s)
I20260812 01:03:43.715338 23855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 01:03:43.720549 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 23855
I20260812 01:03:43.720749 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance
I20260812 01:03:43.721316 22309 external_mini_cluster.cc:1428] Setting key 1a06d6856a7b5989f3a55bff10818778
Aug 12 01:03:43 dist-test-slave-7f01 krb5kdc[23673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496623, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.65@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:03:43.737824 24065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:43.717890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:44159 (local address 127.21.201.126:44891)
0812 01:03:43.718349 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:43.718354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:43.719334 (+ 980us) server_negotiation.cc:407] Connection header received
0812 01:03:43.721813 (+ 2479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:43.721818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:43.721894 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:43.722040 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:43.723823 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:43.724622 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:43.727246 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:43.727667 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:43.727858 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:43.732144 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:43.732149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:43.732153 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:43.732185 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:43.734790 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:43.735512 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:43.735518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:43.735526 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:43.735642 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:43.736090 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:43.736095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:43.736103 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:43.736364 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:43.736521 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:43.737207 (+ 686us) server_negotiation.cc:299] Negotiation successful
0812 01:03:43.737455 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":232,"threads_started":1}
I20260812 01:03:43.738763 24066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:43.718455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44891 (local address 127.21.201.65:44159)
0812 01:03:43.719171 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:43.719212 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:43.721513 (+ 2301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:43.722286 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:43.722296 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:43.722877 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:03:43.723621 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:43.723647 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:43.724784 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:43.724792 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:43.727074 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:43.727093 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:43.727976 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:43.727985 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:43.728316 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:43.729113 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:03:43.729156 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:03:43.731934 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:03:43.734944 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:03:43.734953 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:03:43.734969 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:03:43.735387 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:03:43.735773 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:03:43.735778 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:03:43.735781 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:03:43.735971 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:03:43.736491 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:03:43.736499 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:03:43.736829 (+ 330us) client_negotiation.cc:769] Sending connection context
0812 01:03:43.737099 (+ 270us) client_negotiation.cc:240] Negotiation successful
0812 01:03:43.737610 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":295,"threads_started":1}
I20260812 01:03:43.740407 24062 heartbeater.cc:347] Connected to a master server at 127.21.201.126:44891
I20260812 01:03:43.740734 24062 heartbeater.cc:464] Registering TS with master...
I20260812 01:03:43.741621 24062 heartbeater.cc:511] Master 127.21.201.126:44891 requested a full tablet report, sending...
I20260812 01:03:43.744210 23747 ts_manager.cc:194] Registered new tserver with Master: 2b6d02dc8fd74f8e89e3557ae36cb23a (127.21.201.65:43281)
I20260812 01:03:43.745779 23747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.65@KRBTEST.COM'} at 127.21.201.65:44159
WARNING: no policy specified for kudu/127.21.201.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
Entry for principal kudu/127.21.201.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab.
I20260812 01:03:43.791527 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:0
--local_ip_for_outbound_sockets=127.21.201.66
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:44891
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:44391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:03:44.143292 24071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:44.143580 24071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:44.143800 24071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:44.151016 24071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:44.151146 24071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:44.151317 24071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 01:03:44.161265 24071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:44391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.66.keytab
--principal=kudu/127.21.201.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:44891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:44.162547 24071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:44.163952 24071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:44.177000 24078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:44.176970 24079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:44.177052 24082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:44.177826 24071 server_base.cc:1034] running on GCE node
I20260812 01:03:44.178467 24071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:44.179380 24071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:44.180636 24071 hybrid_clock.cc:648] HybridClock initialized: now 1786496624180587 us; error 51 us; skew 500 ppm
I20260812 01:03:44.180984 24071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:03:44 dist-test-slave-7f01 krb5kdc[23673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496624, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:03:44.184355 24071 init.cc:377] Logged in from keytab as kudu/127.21.201.66@KRBTEST.COM (short username kudu)
I20260812 01:03:44.185734 24071 webserver.cc:460] Webserver started at http://127.21.201.66:34583/ using document root <none> and password file <none>
I20260812 01:03:44.186376 24071 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:44.186479 24071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:44.186750 24071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:44.189743 24071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-1/data/instance:
uuid: "4517a746a774413db805bbc4ace79ea0"
format_stamp: "Formatted at 2026-08-12 01:03:44 on dist-test-slave-7f01"
server_key: "f560db0a6f573e30915a52d07ac43015"
server_key_iv: "33585d46b6432b862427ed733bcb7177"
server_key_version: "encryptionkey@0"
I20260812 01:03:44.190577 24071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance:
uuid: "4517a746a774413db805bbc4ace79ea0"
format_stamp: "Formatted at 2026-08-12 01:03:44 on dist-test-slave-7f01"
server_key: "f560db0a6f573e30915a52d07ac43015"
server_key_iv: "33585d46b6432b862427ed733bcb7177"
server_key_version: "encryptionkey@0"
I20260812 01:03:44.195679 24071 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260812 01:03:44.199317 24091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:44.200620 24071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:03:44.200788 24071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "4517a746a774413db805bbc4ace79ea0"
format_stamp: "Formatted at 2026-08-12 01:03:44 on dist-test-slave-7f01"
server_key: "f560db0a6f573e30915a52d07ac43015"
server_key_iv: "33585d46b6432b862427ed733bcb7177"
server_key_version: "encryptionkey@0"
I20260812 01:03:44.200986 24071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:44.214713 24071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:44.218660 24071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:44.218950 24071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:44.219937 24071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:03:44.221289 24071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:03:44.221365 24071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:44.221534 24071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:03:44.221606 24071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:44.256018 24071 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:42589
I20260812 01:03:44.256073 24274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:42589 every 8 connection(s)
I20260812 01:03:44.257328 24071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 01:03:44.262060 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 24071
I20260812 01:03:44.262321 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance
I20260812 01:03:44.262800 22309 external_mini_cluster.cc:1428] Setting key df4af120457d141abb7078fa50ee1a3f
Aug 12 01:03:44 dist-test-slave-7f01 krb5kdc[23673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496624, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.66@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:03:44.281709 24279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:44.261812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:38509 (local address 127.21.201.126:44891)
0812 01:03:44.262675 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:44.262683 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:44.262710 (+ 27us) server_negotiation.cc:407] Connection header received
0812 01:03:44.264376 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:44.264381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:44.264454 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:44.264541 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:44.266178 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:44.267399 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:44.269097 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:44.269829 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:44.269987 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:44.273808 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:44.273817 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:44.273825 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:44.273874 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:44.277909 (+ 4035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:44.278740 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:44.278748 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:44.278754 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:44.278932 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:44.279503 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:44.279507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:44.279510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:44.279906 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:44.280035 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:44.281194 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0812 01:03:44.281394 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":585,"threads_started":1}
I20260812 01:03:44.283222 24278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:44.261030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44891 (local address 127.21.201.66:38509)
0812 01:03:44.261913 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:44.261962 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:44.264069 (+ 2107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:44.264825 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:44.264837 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:44.265433 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:03:44.265995 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:44.266013 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:44.267577 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:44.267582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:44.268940 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:44.268954 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:44.270082 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:44.270098 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:44.270316 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:44.271141 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:03:44.271164 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:03:44.273540 (+ 2376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:03:44.278096 (+ 4556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:03:44.278104 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:03:44.278117 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:03:44.278590 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:03:44.279051 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:03:44.279054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:03:44.279057 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:03:44.279341 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:03:44.280100 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:03:44.280110 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:03:44.280680 (+ 570us) client_negotiation.cc:769] Sending connection context
0812 01:03:44.281036 (+ 356us) client_negotiation.cc:240] Negotiation successful
0812 01:03:44.281680 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":384,"threads_started":1}
I20260812 01:03:44.284881 24275 heartbeater.cc:347] Connected to a master server at 127.21.201.126:44891
I20260812 01:03:44.285244 24275 heartbeater.cc:464] Registering TS with master...
I20260812 01:03:44.286000 24275 heartbeater.cc:511] Master 127.21.201.126:44891 requested a full tablet report, sending...
I20260812 01:03:44.287559 23747 ts_manager.cc:194] Registered new tserver with Master: 4517a746a774413db805bbc4ace79ea0 (127.21.201.66:42589)
I20260812 01:03:44.288450 23747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.66@KRBTEST.COM'} at 127.21.201.66:38509
WARNING: no policy specified for kudu/127.21.201.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
Entry for principal kudu/127.21.201.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab.
I20260812 01:03:44.335457 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:0
--local_ip_for_outbound_sockets=127.21.201.67
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:44891
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:44391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:03:44.676111 24284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:44.676390 24284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:44.676563 24284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:44.683596 24284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:44.683698 24284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:44.683918 24284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 01:03:44.693488 24284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:44391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.67.keytab
--principal=kudu/127.21.201.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:44891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:44.694772 24284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:44.696156 24284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:44.709306 24297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:44.709549 24301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:44.709645 24303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:44.710363 24284 server_base.cc:1034] running on GCE node
I20260812 01:03:44.711086 24284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:44.712016 24284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:44.713212 24284 hybrid_clock.cc:648] HybridClock initialized: now 1786496624713163 us; error 50 us; skew 500 ppm
I20260812 01:03:44.713616 24284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:03:44 dist-test-slave-7f01 krb5kdc[23673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496624, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:03:44.716861 24284 init.cc:377] Logged in from keytab as kudu/127.21.201.67@KRBTEST.COM (short username kudu)
I20260812 01:03:44.718533 24284 webserver.cc:460] Webserver started at http://127.21.201.67:45655/ using document root <none> and password file <none>
I20260812 01:03:44.719209 24284 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:44.719345 24284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:44.719683 24284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:44.722755 24284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-2/data/instance:
uuid: "c02633f523ea4668b2dbf7cfbed2dfae"
format_stamp: "Formatted at 2026-08-12 01:03:44 on dist-test-slave-7f01"
server_key: "bd7256a75752d438c0b9f49cf632b414"
server_key_iv: "863567ba37fc617af139cc0f08d378e4"
server_key_version: "encryptionkey@0"
I20260812 01:03:44.723623 24284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance:
uuid: "c02633f523ea4668b2dbf7cfbed2dfae"
format_stamp: "Formatted at 2026-08-12 01:03:44 on dist-test-slave-7f01"
server_key: "bd7256a75752d438c0b9f49cf632b414"
server_key_iv: "863567ba37fc617af139cc0f08d378e4"
server_key_version: "encryptionkey@0"
I20260812 01:03:44.729028 24284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260812 01:03:44.732697 24317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:44.734068 24284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:03:44.734251 24284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "c02633f523ea4668b2dbf7cfbed2dfae"
format_stamp: "Formatted at 2026-08-12 01:03:44 on dist-test-slave-7f01"
server_key: "bd7256a75752d438c0b9f49cf632b414"
server_key_iv: "863567ba37fc617af139cc0f08d378e4"
server_key_version: "encryptionkey@0"
I20260812 01:03:44.734477 24284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:44.749471 24062 heartbeater.cc:503] Master 127.21.201.126:44891 was elected leader, sending a full tablet report...
I20260812 01:03:44.760934 24284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:44.765491 24284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:44.765803 24284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:44.766832 24284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:03:44.768299 24284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:03:44.768385 24284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:44.768574 24284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:03:44.768630 24284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:44.804667 24284 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:42375
I20260812 01:03:44.804725 24500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:42375 every 8 connection(s)
I20260812 01:03:44.806191 24284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 01:03:44.815382 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 24284
I20260812 01:03:44.815595 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance
I20260812 01:03:44.816125 22309 external_mini_cluster.cc:1428] Setting key 97587c8d7d78fe12ea93deb6dc189e3e
Aug 12 01:03:44 dist-test-slave-7f01 krb5kdc[23673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496624, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.67@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:03:44.828141 24506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:44.808857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:57977 (local address 127.21.201.126:44891)
0812 01:03:44.809366 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:44.809371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:44.810581 (+ 1210us) server_negotiation.cc:407] Connection header received
0812 01:03:44.812707 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:44.812712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:44.812810 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:44.812964 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:44.814496 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:44.815518 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:44.817468 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:44.818052 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:44.818165 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:44.821959 (+ 3794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:44.821964 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:44.821968 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:44.822004 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:44.825019 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:44.825818 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:44.825822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:44.825825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:44.825957 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:44.826422 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:44.826426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:44.826428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:44.826729 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:44.826879 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:44.827570 (+ 691us) server_negotiation.cc:299] Negotiation successful
0812 01:03:44.827819 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":227,"threads_started":1}
I20260812 01:03:44.829083 24508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:44.809445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:44891 (local address 127.21.201.67:57977)
0812 01:03:44.810401 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:44.810448 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:44.812369 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:44.813207 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:44.813216 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:44.813747 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:03:44.814311 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:44.814329 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:44.815768 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:44.815775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:44.817270 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:44.817283 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:44.818292 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:44.818297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:44.818511 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:44.819294 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:03:44.819325 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:03:44.821745 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:03:44.825174 (+ 3429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:03:44.825182 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:03:44.825195 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:03:44.825668 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:03:44.826078 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:03:44.826082 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:03:44.826084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:03:44.826273 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:03:44.826856 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:03:44.826861 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:03:44.827211 (+ 350us) client_negotiation.cc:769] Sending connection context
0812 01:03:44.827450 (+ 239us) client_negotiation.cc:240] Negotiation successful
0812 01:03:44.827925 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":420,"threads_started":1}
I20260812 01:03:44.830595 24502 heartbeater.cc:347] Connected to a master server at 127.21.201.126:44891
I20260812 01:03:44.830906 24502 heartbeater.cc:464] Registering TS with master...
I20260812 01:03:44.831645 24502 heartbeater.cc:511] Master 127.21.201.126:44891 requested a full tablet report, sending...
I20260812 01:03:44.833101 23745 ts_manager.cc:194] Registered new tserver with Master: c02633f523ea4668b2dbf7cfbed2dfae (127.21.201.67:42375)
I20260812 01:03:44.833985 23745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.201.67@KRBTEST.COM'} at 127.21.201.67:57977
I20260812 01:03:44.843907 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:03:44.861655 24506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:44.849140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50612 (local address 127.21.201.126:44891)
0812 01:03:44.849322 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:44.849326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:44.849769 (+ 443us) server_negotiation.cc:407] Connection header received
0812 01:03:44.849926 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:44.849929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:44.849993 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:44.850076 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:44.851501 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:44.852323 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:44.853573 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:44.853982 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:44.854074 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:44.856501 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:44.856506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:44.856509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:44.856541 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:44.859141 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:44.859848 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:44.859852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:44.859854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:44.859980 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:44.860359 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:44.860362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:44.860364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:44.860701 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:44.860898 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:44.861220 (+ 322us) server_negotiation.cc:299] Negotiation successful
0812 01:03:44.861407 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 01:03:44.865728 23745 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:03:44.872800 23745 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:03:44.917037 24533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:44.889235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32792 (local address 127.21.201.66:42589)
0812 01:03:44.906651 (+ 17416us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:44.906664 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:44.907375 (+ 711us) server_negotiation.cc:407] Connection header received
0812 01:03:44.907689 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:44.907700 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:44.908033 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:44.908216 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:44.909826 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:44.911396 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:44.914624 (+ 3228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:44.916089 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:44.916230 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:44.916414 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:44.916662 (+ 248us) server_negotiation.cc:299] Negotiation successful
0812 01:03:44.916805 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17179,"thread_start_us":17023,"threads_started":1}
I20260812 01:03:44.917135 24530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:44.886072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.21.201.65:43281)
0812 01:03:44.907860 (+ 21788us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:44.907870 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:44.907900 (+ 30us) server_negotiation.cc:407] Connection header received
0812 01:03:44.908001 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:44.908010 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:44.908202 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:44.908409 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:44.909550 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:44.911396 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:44.914563 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:44.915555 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:44.915713 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:44.915994 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:44.916675 (+ 681us) server_negotiation.cc:299] Negotiation successful
0812 01:03:44.916849 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21450,"thread_start_us":21252,"threads_started":1}
I20260812 01:03:44.917773 24525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:44.888526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:42589 (local address 127.0.0.1:32792)
0812 01:03:44.907242 (+ 18716us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:44.907265 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:44.907480 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:44.908512 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:44.908517 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:44.908544 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:44.909618 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:44.909636 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:44.911595 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:44.911599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:44.914455 (+ 2856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:44.914488 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:44.916307 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:44.916313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:44.916477 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:44.916507 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 01:03:44.916611 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 01:03:44.916732 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18540,"mutex_wait_us":149,"thread_start_us":15471,"threads_started":1}
I20260812 01:03:44.918385 24531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:44.886260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36332 (local address 127.21.201.67:42375)
0812 01:03:44.908156 (+ 21896us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:44.908165 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:44.908197 (+ 32us) server_negotiation.cc:407] Connection header received
0812 01:03:44.908303 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:44.908310 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:44.908526 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:44.909211 (+ 685us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:44.909776 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:44.911213 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:44.914583 (+ 3370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:44.916089 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:44.916230 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:44.916450 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:44.917803 (+ 1353us) server_negotiation.cc:299] Negotiation successful
0812 01:03:44.918002 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21686,"thread_start_us":21500,"threads_started":1}
I20260812 01:03:44.920158 24526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:44.885787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:43281 (local address 127.0.0.1:50962)
0812 01:03:44.900386 (+ 14599us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:44.900449 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:44.900719 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:44.908453 (+ 7734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:44.908461 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:44.908495 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:44.909174 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:44.909192 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:44.911548 (+ 2356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:44.911556 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:44.914383 (+ 2827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:44.914402 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:44.915801 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:44.915808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:44.916050 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:44.916099 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 01:03:44.916248 (+ 149us) client_negotiation.cc:240] Negotiation successful
0812 01:03:44.916457 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14218,"mutex_wait_us":121,"thread_start_us":15191,"threads_started":1}
I20260812 01:03:44.920166 24532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:44.885840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:42375 (local address 127.0.0.1:36332)
0812 01:03:44.904387 (+ 18547us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:44.904411 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:44.904552 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:44.908757 (+ 4205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:44.908761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:44.908782 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:44.909527 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:44.909536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:44.911358 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:44.911366 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:44.914404 (+ 3038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:44.914417 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:44.917405 (+ 2988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:44.917421 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:44.917607 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:44.917640 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 01:03:44.917750 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 01:03:44.917912 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18366,"mutex_wait_us":360,"thread_start_us":21025,"threads_started":1}
I20260812 01:03:44.921586 24167 tablet_service.cc:1467] Processing CreateTablet for tablet c9b0e48845ff455796999a62c9c3fc4e (DEFAULT_TABLE table=test-table [id=7c8d524b755d4b49be42a8020c01302a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:03:44.923202 24167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9b0e48845ff455796999a62c9c3fc4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:03:44.923930 24398 tablet_service.cc:1467] Processing CreateTablet for tablet c9b0e48845ff455796999a62c9c3fc4e (DEFAULT_TABLE table=test-table [id=7c8d524b755d4b49be42a8020c01302a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:03:44.925364 24398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9b0e48845ff455796999a62c9c3fc4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:03:44.929617 23954 tablet_service.cc:1467] Processing CreateTablet for tablet c9b0e48845ff455796999a62c9c3fc4e (DEFAULT_TABLE table=test-table [id=7c8d524b755d4b49be42a8020c01302a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:03:44.930735 23954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9b0e48845ff455796999a62c9c3fc4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:03:44.938521 24534 tablet_bootstrap.cc:492] T c9b0e48845ff455796999a62c9c3fc4e P 4517a746a774413db805bbc4ace79ea0: Bootstrap starting.
I20260812 01:03:44.940507 24535 tablet_bootstrap.cc:492] T c9b0e48845ff455796999a62c9c3fc4e P c02633f523ea4668b2dbf7cfbed2dfae: Bootstrap starting.
I20260812 01:03:44.942443 24540 tablet_bootstrap.cc:492] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a: Bootstrap starting.
I20260812 01:03:44.943981 24534 tablet_bootstrap.cc:654] T c9b0e48845ff455796999a62c9c3fc4e P 4517a746a774413db805bbc4ace79ea0: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:44.944146 24535 tablet_bootstrap.cc:654] T c9b0e48845ff455796999a62c9c3fc4e P c02633f523ea4668b2dbf7cfbed2dfae: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:44.945073 24534 log.cc:824] T c9b0e48845ff455796999a62c9c3fc4e P 4517a746a774413db805bbc4ace79ea0: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:44.945399 24535 log.cc:824] T c9b0e48845ff455796999a62c9c3fc4e P c02633f523ea4668b2dbf7cfbed2dfae: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:44.946264 24540 tablet_bootstrap.cc:654] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:44.947495 24540 log.cc:824] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:44.948463 24535 tablet_bootstrap.cc:492] T c9b0e48845ff455796999a62c9c3fc4e P c02633f523ea4668b2dbf7cfbed2dfae: No bootstrap required, opened a new log
I20260812 01:03:44.948503 24534 tablet_bootstrap.cc:492] T c9b0e48845ff455796999a62c9c3fc4e P 4517a746a774413db805bbc4ace79ea0: No bootstrap required, opened a new log
I20260812 01:03:44.948761 24534 ts_tablet_manager.cc:1397] T c9b0e48845ff455796999a62c9c3fc4e P 4517a746a774413db805bbc4ace79ea0: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260812 01:03:44.948789 24535 ts_tablet_manager.cc:1397] T c9b0e48845ff455796999a62c9c3fc4e P c02633f523ea4668b2dbf7cfbed2dfae: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260812 01:03:44.951098 24540 tablet_bootstrap.cc:492] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a: No bootstrap required, opened a new log
I20260812 01:03:44.951377 24540 ts_tablet_manager.cc:1397] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260812 01:03:44.956210 24534 raft_consensus.cc:348] T c9b0e48845ff455796999a62c9c3fc4e P 4517a746a774413db805bbc4ace79ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d02dc8fd74f8e89e3557ae36cb23a" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43281 } } peers { permanent_uuid: "c02633f523ea4668b2dbf7cfbed2dfae" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 42375 } } peers { permanent_uuid: "4517a746a774413db805bbc4ace79ea0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42589 } }
I20260812 01:03:44.956604 24534 raft_consensus.cc:374] T c9b0e48845ff455796999a62c9c3fc4e P 4517a746a774413db805bbc4ace79ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:44.956732 24534 raft_consensus.cc:729] T c9b0e48845ff455796999a62c9c3fc4e P 4517a746a774413db805bbc4ace79ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4517a746a774413db805bbc4ace79ea0, State: Initialized, Role: FOLLOWER
I20260812 01:03:44.957336 24534 consensus_queue.cc:260] T c9b0e48845ff455796999a62c9c3fc4e P 4517a746a774413db805bbc4ace79ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d02dc8fd74f8e89e3557ae36cb23a" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43281 } } peers { permanent_uuid: "c02633f523ea4668b2dbf7cfbed2dfae" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 42375 } } peers { permanent_uuid: "4517a746a774413db805bbc4ace79ea0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42589 } }
I20260812 01:03:44.958454 24275 heartbeater.cc:503] Master 127.21.201.126:44891 was elected leader, sending a full tablet report...
I20260812 01:03:44.959501 24535 raft_consensus.cc:348] T c9b0e48845ff455796999a62c9c3fc4e P c02633f523ea4668b2dbf7cfbed2dfae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d02dc8fd74f8e89e3557ae36cb23a" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43281 } } peers { permanent_uuid: "c02633f523ea4668b2dbf7cfbed2dfae" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 42375 } } peers { permanent_uuid: "4517a746a774413db805bbc4ace79ea0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42589 } }
I20260812 01:03:44.959964 24535 raft_consensus.cc:374] T c9b0e48845ff455796999a62c9c3fc4e P c02633f523ea4668b2dbf7cfbed2dfae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:44.960088 24535 raft_consensus.cc:729] T c9b0e48845ff455796999a62c9c3fc4e P c02633f523ea4668b2dbf7cfbed2dfae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c02633f523ea4668b2dbf7cfbed2dfae, State: Initialized, Role: FOLLOWER
I20260812 01:03:44.960434 24534 ts_tablet_manager.cc:1428] T c9b0e48845ff455796999a62c9c3fc4e P 4517a746a774413db805bbc4ace79ea0: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260812 01:03:44.960659 24535 consensus_queue.cc:260] T c9b0e48845ff455796999a62c9c3fc4e P c02633f523ea4668b2dbf7cfbed2dfae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d02dc8fd74f8e89e3557ae36cb23a" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43281 } } peers { permanent_uuid: "c02633f523ea4668b2dbf7cfbed2dfae" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 42375 } } peers { permanent_uuid: "4517a746a774413db805bbc4ace79ea0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42589 } }
I20260812 01:03:44.961676 24502 heartbeater.cc:503] Master 127.21.201.126:44891 was elected leader, sending a full tablet report...
I20260812 01:03:44.962394 24535 ts_tablet_manager.cc:1428] T c9b0e48845ff455796999a62c9c3fc4e P c02633f523ea4668b2dbf7cfbed2dfae: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260812 01:03:44.964555 24540 raft_consensus.cc:348] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d02dc8fd74f8e89e3557ae36cb23a" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43281 } } peers { permanent_uuid: "c02633f523ea4668b2dbf7cfbed2dfae" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 42375 } } peers { permanent_uuid: "4517a746a774413db805bbc4ace79ea0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42589 } }
I20260812 01:03:44.965003 24540 raft_consensus.cc:374] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:44.965121 24540 raft_consensus.cc:729] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b6d02dc8fd74f8e89e3557ae36cb23a, State: Initialized, Role: FOLLOWER
I20260812 01:03:44.965633 24540 consensus_queue.cc:260] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d02dc8fd74f8e89e3557ae36cb23a" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43281 } } peers { permanent_uuid: "c02633f523ea4668b2dbf7cfbed2dfae" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 42375 } } peers { permanent_uuid: "4517a746a774413db805bbc4ace79ea0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42589 } }
I20260812 01:03:44.967105 24540 ts_tablet_manager.cc:1428] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a: Time spent starting tablet: real 0.016s user 0.010s sys 0.003s
W20260812 01:03:44.968021 24064 tablet.cc:2369] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:03:45.009778 24276 tablet.cc:2369] T c9b0e48845ff455796999a62c9c3fc4e P 4517a746a774413db805bbc4ace79ea0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:03:45.057950 24505 tablet.cc:2369] T c9b0e48845ff455796999a62c9c3fc4e P c02633f523ea4668b2dbf7cfbed2dfae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:03:45.251847 24549 raft_consensus.cc:482] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:03:45.252211 24549 raft_consensus.cc:504] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d02dc8fd74f8e89e3557ae36cb23a" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43281 } } peers { permanent_uuid: "c02633f523ea4668b2dbf7cfbed2dfae" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 42375 } } peers { permanent_uuid: "4517a746a774413db805bbc4ace79ea0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42589 } }
I20260812 01:03:45.254011 24549 leader_election.cc:288] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c02633f523ea4668b2dbf7cfbed2dfae (127.21.201.67:42375), 4517a746a774413db805bbc4ace79ea0 (127.21.201.66:42589)
I20260812 01:03:45.263602 24531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:45.254649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:36925 (local address 127.21.201.67:42375)
0812 01:03:45.254908 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:45.254918 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:45.255951 (+ 1033us) server_negotiation.cc:407] Connection header received
0812 01:03:45.256242 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:45.256250 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:45.256351 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:45.256449 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:45.257217 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:45.257952 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:45.260820 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:45.262145 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:45.262298 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:45.262378 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:45.263192 (+ 814us) server_negotiation.cc:299] Negotiation successful
0812 01:03:45.263341 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 01:03:45.263670 24533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:45.254658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:43839 (local address 127.21.201.66:42589)
0812 01:03:45.254926 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:45.254935 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:45.255851 (+ 916us) server_negotiation.cc:407] Connection header received
0812 01:03:45.256145 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:45.256153 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:45.256244 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:45.256344 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:45.257053 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:45.258220 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:45.260821 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:45.262217 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:45.262324 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:45.262413 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:45.263280 (+ 867us) server_negotiation.cc:299] Negotiation successful
0812 01:03:45.263423 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 01:03:45.264492 24550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:45.254302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:42375 (local address 127.21.201.65:36925)
0812 01:03:45.255845 (+ 1543us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:45.255866 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:45.256005 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:45.256548 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:45.256553 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:45.256577 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:45.257058 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:45.257069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:45.258098 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:45.258103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:45.260616 (+ 2513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:45.260637 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:45.262623 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:45.262635 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:45.262934 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:45.262972 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 01:03:45.263833 (+ 861us) client_negotiation.cc:240] Negotiation successful
0812 01:03:45.263999 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1375,"mutex_wait_us":222,"thread_start_us":1224,"threads_started":1}
I20260812 01:03:45.264534 24552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:45.254302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:42589 (local address 127.21.201.65:43839)
0812 01:03:45.255694 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:45.255717 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:45.255918 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:45.256362 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:45.256367 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:45.256400 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:45.256875 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:45.256890 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:45.258357 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:45.258362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:45.260645 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:45.260655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:45.262652 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:45.262658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:45.262989 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:45.263018 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 01:03:45.263906 (+ 888us) client_negotiation.cc:240] Negotiation successful
0812 01:03:45.264015 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1235,"mutex_wait_us":209,"thread_start_us":946,"threads_started":1}
I20260812 01:03:45.265679 24198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b0e48845ff455796999a62c9c3fc4e" candidate_uuid: "2b6d02dc8fd74f8e89e3557ae36cb23a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4517a746a774413db805bbc4ace79ea0" is_pre_election: true
I20260812 01:03:45.266325 24198 raft_consensus.cc:2454] T c9b0e48845ff455796999a62c9c3fc4e P 4517a746a774413db805bbc4ace79ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b6d02dc8fd74f8e89e3557ae36cb23a in term 0.
I20260812 01:03:45.267149 23882 leader_election.cc:302] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b6d02dc8fd74f8e89e3557ae36cb23a, 4517a746a774413db805bbc4ace79ea0; no voters:
I20260812 01:03:45.267136 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b0e48845ff455796999a62c9c3fc4e" candidate_uuid: "2b6d02dc8fd74f8e89e3557ae36cb23a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c02633f523ea4668b2dbf7cfbed2dfae" is_pre_election: true
I20260812 01:03:45.267647 24427 raft_consensus.cc:2454] T c9b0e48845ff455796999a62c9c3fc4e P c02633f523ea4668b2dbf7cfbed2dfae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b6d02dc8fd74f8e89e3557ae36cb23a in term 0.
I20260812 01:03:45.267757 24549 raft_consensus.cc:2781] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:03:45.267905 24549 raft_consensus.cc:482] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:03:45.267997 24549 raft_consensus.cc:3037] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:45.271423 24549 raft_consensus.cc:504] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d02dc8fd74f8e89e3557ae36cb23a" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43281 } } peers { permanent_uuid: "c02633f523ea4668b2dbf7cfbed2dfae" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 42375 } } peers { permanent_uuid: "4517a746a774413db805bbc4ace79ea0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42589 } }
I20260812 01:03:45.272528 24549 leader_election.cc:288] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a [CANDIDATE]: Term 1 election: Requested vote from peers c02633f523ea4668b2dbf7cfbed2dfae (127.21.201.67:42375), 4517a746a774413db805bbc4ace79ea0 (127.21.201.66:42589)
I20260812 01:03:45.272699 24198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b0e48845ff455796999a62c9c3fc4e" candidate_uuid: "2b6d02dc8fd74f8e89e3557ae36cb23a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4517a746a774413db805bbc4ace79ea0"
I20260812 01:03:45.272836 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b0e48845ff455796999a62c9c3fc4e" candidate_uuid: "2b6d02dc8fd74f8e89e3557ae36cb23a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c02633f523ea4668b2dbf7cfbed2dfae"
I20260812 01:03:45.272931 24198 raft_consensus.cc:3037] T c9b0e48845ff455796999a62c9c3fc4e P 4517a746a774413db805bbc4ace79ea0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:45.273065 24427 raft_consensus.cc:3037] T c9b0e48845ff455796999a62c9c3fc4e P c02633f523ea4668b2dbf7cfbed2dfae [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:45.275316 24427 raft_consensus.cc:2454] T c9b0e48845ff455796999a62c9c3fc4e P c02633f523ea4668b2dbf7cfbed2dfae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b6d02dc8fd74f8e89e3557ae36cb23a in term 1.
I20260812 01:03:45.275313 24198 raft_consensus.cc:2454] T c9b0e48845ff455796999a62c9c3fc4e P 4517a746a774413db805bbc4ace79ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b6d02dc8fd74f8e89e3557ae36cb23a in term 1.
I20260812 01:03:45.275775 23879 leader_election.cc:302] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b6d02dc8fd74f8e89e3557ae36cb23a, c02633f523ea4668b2dbf7cfbed2dfae; no voters:
I20260812 01:03:45.276124 24549 raft_consensus.cc:2781] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:03:45.276597 24549 raft_consensus.cc:686] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a [term 1 LEADER]: Becoming Leader. State: Replica: 2b6d02dc8fd74f8e89e3557ae36cb23a, State: Running, Role: LEADER
I20260812 01:03:45.277087 24549 consensus_queue.cc:237] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d02dc8fd74f8e89e3557ae36cb23a" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43281 } } peers { permanent_uuid: "c02633f523ea4668b2dbf7cfbed2dfae" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 42375 } } peers { permanent_uuid: "4517a746a774413db805bbc4ace79ea0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42589 } }
I20260812 01:03:45.281682 23740 catalog_manager.cc:5495] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b6d02dc8fd74f8e89e3557ae36cb23a (127.21.201.65). New cstate: current_term: 1 leader_uuid: "2b6d02dc8fd74f8e89e3557ae36cb23a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b6d02dc8fd74f8e89e3557ae36cb23a" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 43281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c02633f523ea4668b2dbf7cfbed2dfae" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 42375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4517a746a774413db805bbc4ace79ea0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42589 } health_report { overall_health: UNKNOWN } } }
I20260812 01:03:45.368069 24530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:45.362474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.21.201.65:43281)
0812 01:03:45.362663 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:45.362668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:45.362956 (+ 288us) server_negotiation.cc:407] Connection header received
0812 01:03:45.363143 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:45.363146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:45.363232 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:45.363369 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:03:45.363863 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:45.364604 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:45.366041 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:45.366436 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:45.366525 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:45.366930 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:03:45.367455 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:03:45.367577 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:45.367643 (+ 66us) server_negotiation.cc:299] Negotiation successful
0812 01:03:45.367710 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 01:03:45.376227 24198 raft_consensus.cc:1261] T c9b0e48845ff455796999a62c9c3fc4e P 4517a746a774413db805bbc4ace79ea0 [term 1 FOLLOWER]: Refusing update from remote peer 2b6d02dc8fd74f8e89e3557ae36cb23a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:03:45.376233 24427 raft_consensus.cc:1261] T c9b0e48845ff455796999a62c9c3fc4e P c02633f523ea4668b2dbf7cfbed2dfae [term 1 FOLLOWER]: Refusing update from remote peer 2b6d02dc8fd74f8e89e3557ae36cb23a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:03:45.377195 24554 consensus_queue.cc:1035] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c02633f523ea4668b2dbf7cfbed2dfae" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:03:45.377569 24549 consensus_queue.cc:1035] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4517a746a774413db805bbc4ace79ea0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 42589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:03:45.388852 24558 mvcc.cc:205] Tried to move back new op lower bound from 7317490177530732544 to 7317490177138905088. Current Snapshot: MvccSnapshot[applied={T|T < 7317490177530732544}]
I20260812 01:03:45.408099 23747 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:03:45.408401 23747 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:03:45.412530 23747 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a404298d7e5c4179994243309f373faf: Sending DeleteTablet for 3 replicas of tablet c9b0e48845ff455796999a62c9c3fc4e
I20260812 01:03:45.413815 23954 tablet_service.cc:1514] Processing DeleteTablet for tablet c9b0e48845ff455796999a62c9c3fc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:03:45 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:50962
I20260812 01:03:45.414038 24398 tablet_service.cc:1514] Processing DeleteTablet for tablet c9b0e48845ff455796999a62c9c3fc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:03:45 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:36332
I20260812 01:03:45.415212 24167 tablet_service.cc:1514] Processing DeleteTablet for tablet c9b0e48845ff455796999a62c9c3fc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:03:45 UTC) from {username='kudu', principal='kudu/127.21.201.126@KRBTEST.COM'} at 127.0.0.1:32792
I20260812 01:03:45.416590 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 23855
I20260812 01:03:45.424101 24581 tablet_replica.cc:330] T c9b0e48845ff455796999a62c9c3fc4e P 4517a746a774413db805bbc4ace79ea0: stopping tablet replica
I20260812 01:03:45.424695 24581 raft_consensus.cc:2229] T c9b0e48845ff455796999a62c9c3fc4e P 4517a746a774413db805bbc4ace79ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:03:45.425325 24581 raft_consensus.cc:2258] T c9b0e48845ff455796999a62c9c3fc4e P 4517a746a774413db805bbc4ace79ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:03:45.430124 24581 ts_tablet_manager.cc:1905] T c9b0e48845ff455796999a62c9c3fc4e P 4517a746a774413db805bbc4ace79ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:03:45.438810 24578 tablet_replica.cc:330] T c9b0e48845ff455796999a62c9c3fc4e P c02633f523ea4668b2dbf7cfbed2dfae: stopping tablet replica
I20260812 01:03:45.439551 24578 raft_consensus.cc:2229] T c9b0e48845ff455796999a62c9c3fc4e P c02633f523ea4668b2dbf7cfbed2dfae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:03:45.440156 24578 raft_consensus.cc:2258] T c9b0e48845ff455796999a62c9c3fc4e P c02633f523ea4668b2dbf7cfbed2dfae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:03:45.442561 24577 tablet_replica.cc:330] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a: stopping tablet replica
I20260812 01:03:45.443112 24577 raft_consensus.cc:2229] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:03:45.443457 24578 ts_tablet_manager.cc:1905] T c9b0e48845ff455796999a62c9c3fc4e P c02633f523ea4668b2dbf7cfbed2dfae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:03:45.444263 24581 ts_tablet_manager.cc:1918] T c9b0e48845ff455796999a62c9c3fc4e P 4517a746a774413db805bbc4ace79ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:03:45.444454 24581 log.cc:1196] T c9b0e48845ff455796999a62c9c3fc4e P 4517a746a774413db805bbc4ace79ea0: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/c9b0e48845ff455796999a62c9c3fc4e
Aug 12 01:03:45 dist-test-slave-7f01 krb5kdc[23673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.65@KRBTEST.COM
I20260812 01:03:45.445025 24581 ts_tablet_manager.cc:1939] T c9b0e48845ff455796999a62c9c3fc4e P 4517a746a774413db805bbc4ace79ea0: Deleting consensus metadata
I20260812 01:03:45.445971 24577 raft_consensus.cc:2258] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:03:45.447058 23722 catalog_manager.cc:4841] TS 4517a746a774413db805bbc4ace79ea0 (127.21.201.66:42589): tablet c9b0e48845ff455796999a62c9c3fc4e (table test-table [id=7c8d524b755d4b49be42a8020c01302a]) successfully deleted
I20260812 01:03:45.449612 24577 ts_tablet_manager.cc:1905] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:03:45.450958 24578 ts_tablet_manager.cc:1918] T c9b0e48845ff455796999a62c9c3fc4e P c02633f523ea4668b2dbf7cfbed2dfae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:03:45.451140 24578 log.cc:1196] T c9b0e48845ff455796999a62c9c3fc4e P c02633f523ea4668b2dbf7cfbed2dfae: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/c9b0e48845ff455796999a62c9c3fc4e
I20260812 01:03:45.451725 24578 ts_tablet_manager.cc:1939] T c9b0e48845ff455796999a62c9c3fc4e P c02633f523ea4668b2dbf7cfbed2dfae: Deleting consensus metadata
I20260812 01:03:45.453719 23721 catalog_manager.cc:4841] TS c02633f523ea4668b2dbf7cfbed2dfae (127.21.201.67:42375): tablet c9b0e48845ff455796999a62c9c3fc4e (table test-table [id=7c8d524b755d4b49be42a8020c01302a]) successfully deleted
I20260812 01:03:45.456683 24530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:45.418365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50980 (local address 127.21.201.65:43281)
0812 01:03:45.418601 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:45.418611 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:45.430641 (+ 12030us) server_negotiation.cc:407] Connection header received
0812 01:03:45.430968 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:45.430974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:45.431080 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:45.431165 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:45.433540 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:45.435169 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:45.437100 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:45.437874 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:45.438027 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:45.448330 (+ 10303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:45.448344 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:45.448361 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:45.448426 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:45.453030 (+ 4604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:45.454006 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:45.454017 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:45.454021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:45.454225 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:45.454725 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:45.454730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:45.454734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:45.455271 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:45.455828 (+ 557us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:45.456027 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 01:03:45.456303 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 01:03:45.457592 24052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:45.559055 24577 ts_tablet_manager.cc:1918] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:03:45.559404 24577 log.cc:1196] T c9b0e48845ff455796999a62c9c3fc4e P 2b6d02dc8fd74f8e89e3557ae36cb23a: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786496278741276-22309-0/minicluster-data/ts-0/wal/wals/c9b0e48845ff455796999a62c9c3fc4e
I20260812 01:03:45.559902 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 23855
I20260812 01:03:45.583938 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 24071
W20260812 01:03:45.584049 23723 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.201.65:43281 (error 108)
I20260812 01:03:45.586190 24586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:45.585502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:43281 (local address 127.0.0.1:50996)
0812 01:03:45.585903 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:45.586009 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":196,"threads_started":1}
W20260812 01:03:45.586483 23723 catalog_manager.cc:4584] TS 2b6d02dc8fd74f8e89e3557ae36cb23a (127.21.201.65:43281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c9b0e48845ff455796999a62c9c3fc4e: Network error: Client connection negotiation failed: client connection to 127.21.201.65:43281: connect: Connection refused (error 111)
Aug 12 01:03:45 dist-test-slave-7f01 krb5kdc[23673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.66@KRBTEST.COM
I20260812 01:03:45.600466 24533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:45.584835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32802 (local address 127.21.201.66:42589)
0812 01:03:45.585034 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:45.585041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:45.585149 (+ 108us) server_negotiation.cc:407] Connection header received
0812 01:03:45.585304 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:45.585310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:45.585403 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:45.585491 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:45.587002 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:45.587995 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:45.589261 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:45.589699 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:45.589815 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:45.594251 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:45.594268 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:45.594284 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:45.594325 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:45.597506 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:45.598231 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:45.598239 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:45.598242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:45.598389 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:45.598774 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:45.598779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:45.598782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:45.599182 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:45.599531 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:45.599831 (+ 300us) server_negotiation.cc:299] Negotiation successful
0812 01:03:45.600058 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 01:03:45.601130 24269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:45.640277 24586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:45.639804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:43281 (local address 127.0.0.1:51004)
0812 01:03:45.640027 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:45.640117 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260812 01:03:45.641394 24586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:45.641010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:43281 (local address 127.0.0.1:51010)
0812 01:03:45.641163 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:45.641242 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 01:03:45.693216 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 24071
I20260812 01:03:45.702169 24586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:45.701474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:43281 (local address 127.0.0.1:51018)
0812 01:03:45.701791 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:45.701944 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260812 01:03:45.703449 24586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:45.702979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:43281 (local address 127.0.0.1:51024)
0812 01:03:45.703157 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:45.703247 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 01:03:45.721837 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 24284
Aug 12 01:03:45 dist-test-slave-7f01 krb5kdc[23673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.67@KRBTEST.COM
I20260812 01:03:45.738204 24531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:45.722737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36346 (local address 127.21.201.67:42375)
0812 01:03:45.722960 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:45.722966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:45.722987 (+ 21us) server_negotiation.cc:407] Connection header received
0812 01:03:45.723170 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:45.723175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:45.723244 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:45.723369 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:45.724891 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:45.725615 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:45.726935 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:45.727342 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:45.727474 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:45.732238 (+ 4764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:45.732255 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:45.732271 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:45.732308 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:45.735086 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:45.735985 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:45.735993 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:45.735996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:45.736138 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:45.736538 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:45.736541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:45.736544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:45.736931 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:45.737248 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:45.737554 (+ 306us) server_negotiation.cc:299] Negotiation successful
0812 01:03:45.737802 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 01:03:45.738897 24495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:45.784296 24586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:45.783599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:43281 (local address 127.0.0.1:51030)
0812 01:03:45.783952 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:45.784088 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260812 01:03:45.785494 24586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:45.785028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:43281 (local address 127.0.0.1:51046)
0812 01:03:45.785240 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:45.785328 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260812 01:03:45.832183 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 24284
I20260812 01:03:45.855374 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 23696
I20260812 01:03:45.856556 23810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:45.976991 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 23696
2026-08-12T01:03:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (5122 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-12T01:03:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:03:46Z Disabled control of system clock
I20260812 01:03:46.082486 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:36781
--webserver_interface=127.21.201.126
--webserver_port=0
--builtin_ntp_servers=127.21.201.84:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:36781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 01:03:46.433024 24601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:46.433319 24601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:46.433485 24601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:46.440721 24601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:03:46.440828 24601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:46.440902 24601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:46.440989 24601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:03:46.441052 24601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:03:46.450042 24601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:36781
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:36781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:46.451329 24601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:46.453140 24601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:46.464920 24611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:46.464989 24617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:46.464964 24612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:46.465890 24601 server_base.cc:1034] running on GCE node
I20260812 01:03:46.466660 24601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:46.468098 24601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:46.469323 24601 hybrid_clock.cc:648] HybridClock initialized: now 1786496626469279 us; error 28 us; skew 500 ppm
I20260812 01:03:46.469702 24601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:03:46.472088 24601 webserver.cc:460] Webserver started at http://127.21.201.126:36099/ using document root <none> and password file <none>
I20260812 01:03:46.472766 24601 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:46.472878 24601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:46.473203 24601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:46.476179 24601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "04f18efa67194fcfa1fb65462c0c0c1e"
format_stamp: "Formatted at 2026-08-12 01:03:46 on dist-test-slave-7f01"
server_key: "a560ba1ff12738271706e6334401cc66"
server_key_iv: "5f3a3a94bcde78e8070b94ab6210b0d1"
server_key_version: "encryptionkey@0"
I20260812 01:03:46.477064 24601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "04f18efa67194fcfa1fb65462c0c0c1e"
format_stamp: "Formatted at 2026-08-12 01:03:46 on dist-test-slave-7f01"
server_key: "a560ba1ff12738271706e6334401cc66"
server_key_iv: "5f3a3a94bcde78e8070b94ab6210b0d1"
server_key_version: "encryptionkey@0"
I20260812 01:03:46.482422 24601 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:03:46.486018 24632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:46.487336 24601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260812 01:03:46.487550 24601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "04f18efa67194fcfa1fb65462c0c0c1e"
format_stamp: "Formatted at 2026-08-12 01:03:46 on dist-test-slave-7f01"
server_key: "a560ba1ff12738271706e6334401cc66"
server_key_iv: "5f3a3a94bcde78e8070b94ab6210b0d1"
server_key_version: "encryptionkey@0"
I20260812 01:03:46.487841 24601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:46.502243 24601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:46.503077 24601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:46.503373 24601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:46.522910 24601 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:36781
I20260812 01:03:46.522948 24703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:36781 every 8 connection(s)
I20260812 01:03:46.524312 24601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:03:46.528193 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:03:46.530892 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 24601
I20260812 01:03:46.531082 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:03:46.531661 22309 external_mini_cluster.cc:1428] Setting key 8f4a9035db0d120d3d2ccc196e2be64c
I20260812 01:03:46.540647 24706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04f18efa67194fcfa1fb65462c0c0c1e: Bootstrap starting.
I20260812 01:03:46.544385 24706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04f18efa67194fcfa1fb65462c0c0c1e: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:46.545567 24706 log.cc:824] T 00000000000000000000000000000000 P 04f18efa67194fcfa1fb65462c0c0c1e: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:46.546103 24712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:46.533705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39384 (local address 127.21.201.126:36781)
0812 01:03:46.534807 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:46.534818 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:46.534862 (+ 44us) server_negotiation.cc:407] Connection header received
0812 01:03:46.536694 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:46.536718 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:46.537185 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:46.537516 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:46.538169 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:46.539504 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:46.541403 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:46.542107 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:46.542252 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:46.543619 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:46.543626 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:46.543638 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:46.544254 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:46.544348 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:46.544371 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:46.544522 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:46.544737 (+ 215us) server_negotiation.cc:299] Negotiation successful
0812 01:03:46.545074 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":449,"threads_started":1}
I20260812 01:03:46.548658 24706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04f18efa67194fcfa1fb65462c0c0c1e: No bootstrap required, opened a new log
I20260812 01:03:46.556123 24706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04f18efa67194fcfa1fb65462c0c0c1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f18efa67194fcfa1fb65462c0c0c1e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 36781 } }
I20260812 01:03:46.556457 24706 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04f18efa67194fcfa1fb65462c0c0c1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:46.556538 24706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04f18efa67194fcfa1fb65462c0c0c1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f18efa67194fcfa1fb65462c0c0c1e, State: Initialized, Role: FOLLOWER
I20260812 01:03:46.557063 24706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04f18efa67194fcfa1fb65462c0c0c1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f18efa67194fcfa1fb65462c0c0c1e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 36781 } }
I20260812 01:03:46.557281 24706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04f18efa67194fcfa1fb65462c0c0c1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:03:46.557379 24706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04f18efa67194fcfa1fb65462c0c0c1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:03:46.557504 24706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04f18efa67194fcfa1fb65462c0c0c1e [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:46.559562 24706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04f18efa67194fcfa1fb65462c0c0c1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f18efa67194fcfa1fb65462c0c0c1e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 36781 } }
I20260812 01:03:46.560127 24706 leader_election.cc:302] T 00000000000000000000000000000000 P 04f18efa67194fcfa1fb65462c0c0c1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04f18efa67194fcfa1fb65462c0c0c1e; no voters:
I20260812 01:03:46.560760 24706 leader_election.cc:288] T 00000000000000000000000000000000 P 04f18efa67194fcfa1fb65462c0c0c1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:03:46.560938 24714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04f18efa67194fcfa1fb65462c0c0c1e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:03:46.561718 24714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04f18efa67194fcfa1fb65462c0c0c1e [term 1 LEADER]: Becoming Leader. State: Replica: 04f18efa67194fcfa1fb65462c0c0c1e, State: Running, Role: LEADER
I20260812 01:03:46.562208 24714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04f18efa67194fcfa1fb65462c0c0c1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f18efa67194fcfa1fb65462c0c0c1e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 36781 } }
I20260812 01:03:46.562871 24706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04f18efa67194fcfa1fb65462c0c0c1e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:03:46.565557 24721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04f18efa67194fcfa1fb65462c0c0c1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04f18efa67194fcfa1fb65462c0c0c1e. Latest consensus state: current_term: 1 leader_uuid: "04f18efa67194fcfa1fb65462c0c0c1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f18efa67194fcfa1fb65462c0c0c1e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 36781 } } }
I20260812 01:03:46.565526 24720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04f18efa67194fcfa1fb65462c0c0c1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04f18efa67194fcfa1fb65462c0c0c1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f18efa67194fcfa1fb65462c0c0c1e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 36781 } } }
I20260812 01:03:46.566077 24720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04f18efa67194fcfa1fb65462c0c0c1e [sys.catalog]: This master's current role is: LEADER
I20260812 01:03:46.566073 24721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04f18efa67194fcfa1fb65462c0c0c1e [sys.catalog]: This master's current role is: LEADER
I20260812 01:03:46.568248 24725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:03:46.573474 24725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:03:46.583107 24725 catalog_manager.cc:1347] Generated new cluster ID: daa40abe062246f886be0a26df9d7031
I20260812 01:03:46.583238 24725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:03:46.620307 24725 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:03:46.621691 24725 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:03:46.630379 24725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04f18efa67194fcfa1fb65462c0c0c1e: Generated new TSK 0
I20260812 01:03:46.631171 24725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:03:46.642181 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.65:0
--local_ip_for_outbound_sockets=127.21.201.65
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:36781
--builtin_ntp_servers=127.21.201.84:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 01:03:47.022877 24750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:47.023160 24750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:47.023250 24750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:47.030577 24750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:47.030675 24750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:47.030792 24750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.65
I20260812 01:03:47.039896 24750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.201.65
--webserver_port=0
--tserver_master_addrs=127.21.201.126:36781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.65
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:47.041209 24750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:47.042948 24750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:47.056460 24761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:47.056509 24762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:47.056533 24766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:47.057642 24750 server_base.cc:1034] running on GCE node
I20260812 01:03:47.058326 24750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:47.059356 24750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:47.060618 24750 hybrid_clock.cc:648] HybridClock initialized: now 1786496627060578 us; error 60 us; skew 500 ppm
I20260812 01:03:47.060961 24750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:03:47.063306 24750 webserver.cc:460] Webserver started at http://127.21.201.65:41009/ using document root <none> and password file <none>
I20260812 01:03:47.064101 24750 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:47.064230 24750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:47.064591 24750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:47.067404 24750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-0/data/instance:
uuid: "e9bdf277f8084e8e9a982268f5fb245a"
format_stamp: "Formatted at 2026-08-12 01:03:47 on dist-test-slave-7f01"
server_key: "121d1a93a98166f41d09d83ed7bd4b92"
server_key_iv: "2c11ae3f3b0237bee4f22b30bc887290"
server_key_version: "encryptionkey@0"
I20260812 01:03:47.068286 24750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance:
uuid: "e9bdf277f8084e8e9a982268f5fb245a"
format_stamp: "Formatted at 2026-08-12 01:03:47 on dist-test-slave-7f01"
server_key: "121d1a93a98166f41d09d83ed7bd4b92"
server_key_iv: "2c11ae3f3b0237bee4f22b30bc887290"
server_key_version: "encryptionkey@0"
I20260812 01:03:47.073490 24750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:03:47.076972 24779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:47.078289 24750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 01:03:47.078435 24750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-0/wal
uuid: "e9bdf277f8084e8e9a982268f5fb245a"
format_stamp: "Formatted at 2026-08-12 01:03:47 on dist-test-slave-7f01"
server_key: "121d1a93a98166f41d09d83ed7bd4b92"
server_key_iv: "2c11ae3f3b0237bee4f22b30bc887290"
server_key_version: "encryptionkey@0"
I20260812 01:03:47.078624 24750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:47.101238 24750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:47.102017 24750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:47.102286 24750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:47.103205 24750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:03:47.104806 24750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:03:47.104897 24750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:47.104980 24750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:03:47.105041 24750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:47.139819 24750 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.65:35453
I20260812 01:03:47.139920 24981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.65:35453 every 8 connection(s)
I20260812 01:03:47.141152 24750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-0/data/info.pb
I20260812 01:03:47.150799 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 24750
I20260812 01:03:47.150991 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-0/wal/instance
I20260812 01:03:47.151567 22309 external_mini_cluster.cc:1428] Setting key 383730b983ab4cde3723f214fd9761b8
I20260812 01:03:47.154892 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.66:0
--local_ip_for_outbound_sockets=127.21.201.66
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:36781
--builtin_ntp_servers=127.21.201.84:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 01:03:47.154928 24985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:47.143920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:58893 (local address 127.21.201.126:36781)
0812 01:03:47.144402 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:47.144406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:47.145626 (+ 1220us) server_negotiation.cc:407] Connection header received
0812 01:03:47.147502 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:47.147508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:47.147600 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:47.147751 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:47.148857 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:47.149616 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:47.151721 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:47.152204 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:47.152309 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:47.153906 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:47.153914 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:47.153919 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:47.153944 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:47.154001 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:47.154026 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:47.154128 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:47.154383 (+ 255us) server_negotiation.cc:299] Negotiation successful
0812 01:03:47.154493 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":249,"threads_started":1}
I20260812 01:03:47.155839 24987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:47.144503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:36781 (local address 127.21.201.65:58893)
0812 01:03:47.145455 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:47.145492 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:47.147228 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:47.148016 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:47.148024 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:47.148112 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:03:47.148656 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:47.148672 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:47.149776 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:47.149787 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:47.151523 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:47.151538 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:47.152461 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:47.152468 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:47.152709 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:47.153499 (+ 790us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:03:47.153503 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:03:47.153558 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:03:47.153662 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:03:47.154157 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:03:47.154167 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:03:47.154174 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 01:03:47.154331 (+ 157us) client_negotiation.cc:240] Negotiation successful
0812 01:03:47.154631 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":314,"threads_started":1}
I20260812 01:03:47.157379 24983 heartbeater.cc:347] Connected to a master server at 127.21.201.126:36781
I20260812 01:03:47.157692 24983 heartbeater.cc:464] Registering TS with master...
I20260812 01:03:47.158454 24983 heartbeater.cc:511] Master 127.21.201.126:36781 requested a full tablet report, sending...
I20260812 01:03:47.160518 24655 ts_manager.cc:194] Registered new tserver with Master: e9bdf277f8084e8e9a982268f5fb245a (127.21.201.65:35453)
I20260812 01:03:47.161979 24655 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.201.65:58893
W20260812 01:03:47.502542 24988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:47.502888 24988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:47.503059 24988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:47.510601 24988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:47.510757 24988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:47.510952 24988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.66
I20260812 01:03:47.520406 24988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.201.66
--webserver_port=0
--tserver_master_addrs=127.21.201.126:36781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.66
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:47.522159 24988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:47.524680 24988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:47.547982 25002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:47.548058 25004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:47.548190 25012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:47.548761 24988 server_base.cc:1034] running on GCE node
I20260812 01:03:47.549400 24988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:47.550362 24988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:47.551594 24988 hybrid_clock.cc:648] HybridClock initialized: now 1786496627551546 us; error 55 us; skew 500 ppm
I20260812 01:03:47.551976 24988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:03:47.554176 24988 webserver.cc:460] Webserver started at http://127.21.201.66:43125/ using document root <none> and password file <none>
I20260812 01:03:47.554874 24988 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:47.555007 24988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:47.555332 24988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:47.558254 24988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-1/data/instance:
uuid: "1093ae1b58084a85b5ff97511c67c2f0"
format_stamp: "Formatted at 2026-08-12 01:03:47 on dist-test-slave-7f01"
server_key: "c4a25f8376df87efc8930c52e1581520"
server_key_iv: "40c9571e9bf0d3e899484383c21b0109"
server_key_version: "encryptionkey@0"
I20260812 01:03:47.559067 24988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance:
uuid: "1093ae1b58084a85b5ff97511c67c2f0"
format_stamp: "Formatted at 2026-08-12 01:03:47 on dist-test-slave-7f01"
server_key: "c4a25f8376df87efc8930c52e1581520"
server_key_iv: "40c9571e9bf0d3e899484383c21b0109"
server_key_version: "encryptionkey@0"
I20260812 01:03:47.564174 24988 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260812 01:03:47.567548 25021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:47.569033 24988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:03:47.569161 24988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-1/wal
uuid: "1093ae1b58084a85b5ff97511c67c2f0"
format_stamp: "Formatted at 2026-08-12 01:03:47 on dist-test-slave-7f01"
server_key: "c4a25f8376df87efc8930c52e1581520"
server_key_iv: "40c9571e9bf0d3e899484383c21b0109"
server_key_version: "encryptionkey@0"
I20260812 01:03:47.569340 24988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:47.582796 24988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:47.583575 24988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:47.583899 24988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:47.584810 24988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:03:47.586186 24988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:03:47.586274 24988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:47.586357 24988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:03:47.586413 24988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:47.621673 24988 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.66:44735
I20260812 01:03:47.621724 25224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.66:44735 every 8 connection(s)
I20260812 01:03:47.623025 24988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-1/data/info.pb
I20260812 01:03:47.623531 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 24988
I20260812 01:03:47.623728 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-1/wal/instance
I20260812 01:03:47.624421 22309 external_mini_cluster.cc:1428] Setting key ee8875a95cf5adc5e2b92678cb723f0a
I20260812 01:03:47.627650 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.201.67:0
--local_ip_for_outbound_sockets=127.21.201.67
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:36781
--builtin_ntp_servers=127.21.201.84:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 01:03:47.636650 24985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:47.625914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.66:50665 (local address 127.21.201.126:36781)
0812 01:03:47.626096 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:47.626100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:47.627914 (+ 1814us) server_negotiation.cc:407] Connection header received
0812 01:03:47.629737 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:47.629745 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:47.629839 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:47.629952 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:47.630970 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:47.632135 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:47.633789 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:47.634221 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:47.634361 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:47.635802 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:47.635808 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:47.635811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:47.635831 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:47.635864 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:47.635876 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:47.635972 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:47.636194 (+ 222us) server_negotiation.cc:299] Negotiation successful
0812 01:03:47.636301 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 01:03:47.637657 25231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:47.626607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:36781 (local address 127.21.201.66:50665)
0812 01:03:47.627755 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:47.627800 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:47.629471 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:47.630139 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:47.630146 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:47.630225 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:03:47.630799 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:47.630815 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:47.632320 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:47.632324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:47.633618 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:47.633631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:47.634469 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:47.634478 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:47.634713 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:47.635462 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:03:47.635465 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:03:47.635503 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:03:47.635577 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:03:47.635974 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:03:47.635988 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:03:47.635997 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 01:03:47.636176 (+ 179us) client_negotiation.cc:240] Negotiation successful
0812 01:03:47.636564 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":386,"threads_started":1}
I20260812 01:03:47.639122 25226 heartbeater.cc:347] Connected to a master server at 127.21.201.126:36781
I20260812 01:03:47.639464 25226 heartbeater.cc:464] Registering TS with master...
I20260812 01:03:47.640379 25226 heartbeater.cc:511] Master 127.21.201.126:36781 requested a full tablet report, sending...
I20260812 01:03:47.642015 24655 ts_manager.cc:194] Registered new tserver with Master: 1093ae1b58084a85b5ff97511c67c2f0 (127.21.201.66:44735)
I20260812 01:03:47.642889 24655 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.201.66:50665
W20260812 01:03:47.977945 25232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:47.978274 25232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:47.978434 25232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:47.985644 25232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:47.985775 25232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:47.985955 25232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.201.67
I20260812 01:03:47.995453 25232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.201.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.201.67
--webserver_port=0
--tserver_master_addrs=127.21.201.126:36781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.201.67
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:47.996809 25232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:47.998620 25232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:48.011940 25241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:48.012104 25239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:48.012076 25237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:48.013053 25232 server_base.cc:1034] running on GCE node
I20260812 01:03:48.013592 25232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:48.014467 25232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:48.015673 25232 hybrid_clock.cc:648] HybridClock initialized: now 1786496628015619 us; error 44 us; skew 500 ppm
I20260812 01:03:48.016036 25232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 01:03:48.018353 25232 webserver.cc:460] Webserver started at http://127.21.201.67:43493/ using document root <none> and password file <none>
I20260812 01:03:48.018981 25232 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:48.019068 25232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:48.019323 25232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:48.022187 25232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-2/data/instance:
uuid: "0031233573a748cd8b2fb9e1039e38e6"
format_stamp: "Formatted at 2026-08-12 01:03:48 on dist-test-slave-7f01"
server_key: "54203cb3f1b1fa444d8a08015dd255b4"
server_key_iv: "88e6d554e5e090f63fd9bd8822dfb14b"
server_key_version: "encryptionkey@0"
I20260812 01:03:48.022926 25232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance:
uuid: "0031233573a748cd8b2fb9e1039e38e6"
format_stamp: "Formatted at 2026-08-12 01:03:48 on dist-test-slave-7f01"
server_key: "54203cb3f1b1fa444d8a08015dd255b4"
server_key_iv: "88e6d554e5e090f63fd9bd8822dfb14b"
server_key_version: "encryptionkey@0"
I20260812 01:03:48.028093 25232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260812 01:03:48.031348 25250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:48.032648 25232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:03:48.032823 25232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-2/wal
uuid: "0031233573a748cd8b2fb9e1039e38e6"
format_stamp: "Formatted at 2026-08-12 01:03:48 on dist-test-slave-7f01"
server_key: "54203cb3f1b1fa444d8a08015dd255b4"
server_key_iv: "88e6d554e5e090f63fd9bd8822dfb14b"
server_key_version: "encryptionkey@0"
I20260812 01:03:48.033028 25232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:48.046629 25232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:48.047417 25232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:48.047722 25232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:48.048789 25232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 01:03:48.050266 25232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 01:03:48.050377 25232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:48.050504 25232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 01:03:48.050592 25232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:48.084335 25232 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.67:46573
I20260812 01:03:48.084376 25455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.67:46573 every 8 connection(s)
I20260812 01:03:48.085675 25232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-2/data/info.pb
I20260812 01:03:48.090741 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 25232
I20260812 01:03:48.091032 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-2/wal/instance
I20260812 01:03:48.091511 22309 external_mini_cluster.cc:1428] Setting key 7e0a1699db9bd06e67a0222b77f87f9e
I20260812 01:03:48.098240 24985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:48.088534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.67:43649 (local address 127.21.201.126:36781)
0812 01:03:48.088723 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:48.088728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:48.090012 (+ 1284us) server_negotiation.cc:407] Connection header received
0812 01:03:48.092031 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:48.092038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:48.092141 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:48.092252 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:48.093280 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.094131 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.095790 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.096263 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.096359 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:48.097565 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:48.097569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:48.097573 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:48.097590 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:48.097622 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:48.097634 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:48.097704 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:48.097905 (+ 201us) server_negotiation.cc:299] Negotiation successful
0812 01:03:48.097980 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 01:03:48.099179 25459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:48.088975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:36781 (local address 127.21.201.67:43649)
0812 01:03:48.089848 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:48.089892 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:48.091752 (+ 1860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:48.092486 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:48.092493 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:48.092574 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:03:48.093116 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:48.093131 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.094328 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.094334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:48.095612 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:48.095624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.096458 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.096463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:48.096655 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:48.097279 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 01:03:48.097282 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:03:48.097318 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 01:03:48.097386 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:03:48.097715 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:03:48.097722 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:03:48.097727 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 01:03:48.097859 (+ 132us) client_negotiation.cc:240] Negotiation successful
0812 01:03:48.098127 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":283,"threads_started":1}
I20260812 01:03:48.100550 25456 heartbeater.cc:347] Connected to a master server at 127.21.201.126:36781
I20260812 01:03:48.100852 25456 heartbeater.cc:464] Registering TS with master...
I20260812 01:03:48.101591 25456 heartbeater.cc:511] Master 127.21.201.126:36781 requested a full tablet report, sending...
I20260812 01:03:48.103024 24655 ts_manager.cc:194] Registered new tserver with Master: 0031233573a748cd8b2fb9e1039e38e6 (127.21.201.67:46573)
I20260812 01:03:48.103907 24655 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.201.67:43649
I20260812 01:03:48.106665 22309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 01:03:48.116644 24985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:48.110993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36050 (local address 127.21.201.126:36781)
0812 01:03:48.111160 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:48.111165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:48.111702 (+ 537us) server_negotiation.cc:407] Connection header received
0812 01:03:48.111921 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:48.111924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:48.111982 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:48.112059 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:48.112556 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.113216 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.114569 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.114964 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.115066 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:48.116100 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:48.116104 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:48.116107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:48.116124 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:48.116154 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:48.116165 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:48.116236 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:48.116350 (+ 114us) server_negotiation.cc:299] Negotiation successful
0812 01:03:48.116425 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 01:03:48.120225 24655 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 01:03:48.127076 24655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 01:03:48.159157 25494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:48.142313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45000 (local address 127.21.201.65:35453)
0812 01:03:48.150053 (+ 7740us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:48.150065 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:48.151821 (+ 1756us) server_negotiation.cc:407] Connection header received
0812 01:03:48.151928 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:48.151934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:48.152121 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:48.152297 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:48.152826 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.154189 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.156317 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.157933 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.158074 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:48.158354 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:48.158630 (+ 276us) server_negotiation.cc:299] Negotiation successful
0812 01:03:48.158791 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7515,"thread_start_us":7396,"threads_started":1}
I20260812 01:03:48.159192 25475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:48.141698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:35453 (local address 127.0.0.1:45000)
0812 01:03:48.150561 (+ 8863us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:48.150584 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:48.150732 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:48.152292 (+ 1560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:48.152297 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:48.152321 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:48.152671 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:48.152681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.154364 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.154368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:48.156127 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:48.156145 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.158144 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.158150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:48.158358 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:48.158424 (+ 66us) client_negotiation.cc:769] Sending connection context
0812 01:03:48.158559 (+ 135us) client_negotiation.cc:240] Negotiation successful
0812 01:03:48.158746 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8702,"mutex_wait_us":43,"thread_start_us":6488,"threads_started":1}
I20260812 01:03:48.159502 25488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:48.139537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43442 (local address 127.21.201.66:44735)
0812 01:03:48.150341 (+ 10804us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:48.150352 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:48.150385 (+ 33us) server_negotiation.cc:407] Connection header received
0812 01:03:48.150538 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:48.150545 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:48.150784 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:48.151017 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:48.151997 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.153589 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.156587 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.157933 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.158073 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:48.158314 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:48.158954 (+ 640us) server_negotiation.cc:299] Negotiation successful
0812 01:03:48.159134 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10660,"thread_start_us":10543,"threads_started":1}
I20260812 01:03:48.159612 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:48.139097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:44735 (local address 127.0.0.1:43442)
0812 01:03:48.148577 (+ 9480us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:48.148619 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:48.148787 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:48.151356 (+ 2569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:48.151363 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:48.151395 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:48.151723 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:48.151761 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.153756 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.153761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:48.156391 (+ 2630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:48.156406 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.158488 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.158495 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:48.158726 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:48.158763 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 01:03:48.158846 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 01:03:48.158977 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9236,"mutex_wait_us":47,"thread_start_us":10226,"threads_started":1}
I20260812 01:03:48.160310 25484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:48.139150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:46573 (local address 127.0.0.1:54184)
0812 01:03:48.149668 (+ 10518us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:48.149690 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:48.149831 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:48.151773 (+ 1942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:48.151780 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:48.151813 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:48.152178 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:48.152193 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.154146 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.154151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:48.156648 (+ 2497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:48.156658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.158488 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.158495 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:48.159891 (+ 1396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:48.159928 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 01:03:48.160011 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 01:03:48.160093 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10342,"mutex_wait_us":1203,"thread_start_us":11121,"threads_started":1}
I20260812 01:03:48.160645 25487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:48.139412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54184 (local address 127.21.201.67:46573)
0812 01:03:48.151342 (+ 11930us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:48.151348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:48.151373 (+ 25us) server_negotiation.cc:407] Connection header received
0812 01:03:48.151450 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:48.151455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:48.151591 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:48.151730 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:48.152409 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.153972 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.156817 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.158212 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.158356 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:48.158617 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:48.160114 (+ 1497us) server_negotiation.cc:299] Negotiation successful
0812 01:03:48.160282 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11771,"thread_start_us":11599,"threads_started":1}
I20260812 01:03:48.163461 25102 tablet_service.cc:1467] Processing CreateTablet for tablet 33fbc39f37314f90906e710d7bdc5abc (DEFAULT_TABLE table=test-table [id=e63e3d47fc2444d792082b44a52d76a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:03:48.163945 25339 tablet_service.cc:1467] Processing CreateTablet for tablet 33fbc39f37314f90906e710d7bdc5abc (DEFAULT_TABLE table=test-table [id=e63e3d47fc2444d792082b44a52d76a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:03:48.163945 24853 tablet_service.cc:1467] Processing CreateTablet for tablet 33fbc39f37314f90906e710d7bdc5abc (DEFAULT_TABLE table=test-table [id=e63e3d47fc2444d792082b44a52d76a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 01:03:48.164973 24983 heartbeater.cc:503] Master 127.21.201.126:36781 was elected leader, sending a full tablet report...
I20260812 01:03:48.165038 25102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33fbc39f37314f90906e710d7bdc5abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:03:48.165272 24853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33fbc39f37314f90906e710d7bdc5abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:03:48.165297 25339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33fbc39f37314f90906e710d7bdc5abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:03:48.178346 25498 tablet_bootstrap.cc:492] T 33fbc39f37314f90906e710d7bdc5abc P 0031233573a748cd8b2fb9e1039e38e6: Bootstrap starting.
I20260812 01:03:48.180635 25499 tablet_bootstrap.cc:492] T 33fbc39f37314f90906e710d7bdc5abc P 1093ae1b58084a85b5ff97511c67c2f0: Bootstrap starting.
I20260812 01:03:48.182201 25501 tablet_bootstrap.cc:492] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a: Bootstrap starting.
I20260812 01:03:48.183897 25498 tablet_bootstrap.cc:654] T 33fbc39f37314f90906e710d7bdc5abc P 0031233573a748cd8b2fb9e1039e38e6: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:48.184259 25499 tablet_bootstrap.cc:654] T 33fbc39f37314f90906e710d7bdc5abc P 1093ae1b58084a85b5ff97511c67c2f0: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:48.185142 25501 tablet_bootstrap.cc:654] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:48.185292 25498 log.cc:824] T 33fbc39f37314f90906e710d7bdc5abc P 0031233573a748cd8b2fb9e1039e38e6: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:48.185393 25499 log.cc:824] T 33fbc39f37314f90906e710d7bdc5abc P 1093ae1b58084a85b5ff97511c67c2f0: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:48.186329 25501 log.cc:824] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:48.188687 25498 tablet_bootstrap.cc:492] T 33fbc39f37314f90906e710d7bdc5abc P 0031233573a748cd8b2fb9e1039e38e6: No bootstrap required, opened a new log
I20260812 01:03:48.188968 25498 ts_tablet_manager.cc:1397] T 33fbc39f37314f90906e710d7bdc5abc P 0031233573a748cd8b2fb9e1039e38e6: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260812 01:03:48.189128 25501 tablet_bootstrap.cc:492] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a: No bootstrap required, opened a new log
I20260812 01:03:48.189307 25499 tablet_bootstrap.cc:492] T 33fbc39f37314f90906e710d7bdc5abc P 1093ae1b58084a85b5ff97511c67c2f0: No bootstrap required, opened a new log
I20260812 01:03:48.189384 25501 ts_tablet_manager.cc:1397] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260812 01:03:48.189548 25499 ts_tablet_manager.cc:1397] T 33fbc39f37314f90906e710d7bdc5abc P 1093ae1b58084a85b5ff97511c67c2f0: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260812 01:03:48.196617 25501 raft_consensus.cc:348] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1093ae1b58084a85b5ff97511c67c2f0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 44735 } } peers { permanent_uuid: "0031233573a748cd8b2fb9e1039e38e6" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 46573 } } peers { permanent_uuid: "e9bdf277f8084e8e9a982268f5fb245a" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35453 } }
I20260812 01:03:48.197010 25501 raft_consensus.cc:374] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:48.197142 25501 raft_consensus.cc:729] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9bdf277f8084e8e9a982268f5fb245a, State: Initialized, Role: FOLLOWER
I20260812 01:03:48.197746 25501 consensus_queue.cc:260] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1093ae1b58084a85b5ff97511c67c2f0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 44735 } } peers { permanent_uuid: "0031233573a748cd8b2fb9e1039e38e6" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 46573 } } peers { permanent_uuid: "e9bdf277f8084e8e9a982268f5fb245a" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35453 } }
I20260812 01:03:48.199612 25501 ts_tablet_manager.cc:1428] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a: Time spent starting tablet: real 0.010s user 0.003s sys 0.007s
I20260812 01:03:48.199663 25498 raft_consensus.cc:348] T 33fbc39f37314f90906e710d7bdc5abc P 0031233573a748cd8b2fb9e1039e38e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1093ae1b58084a85b5ff97511c67c2f0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 44735 } } peers { permanent_uuid: "0031233573a748cd8b2fb9e1039e38e6" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 46573 } } peers { permanent_uuid: "e9bdf277f8084e8e9a982268f5fb245a" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35453 } }
I20260812 01:03:48.200127 25498 raft_consensus.cc:374] T 33fbc39f37314f90906e710d7bdc5abc P 0031233573a748cd8b2fb9e1039e38e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:48.200237 25498 raft_consensus.cc:729] T 33fbc39f37314f90906e710d7bdc5abc P 0031233573a748cd8b2fb9e1039e38e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0031233573a748cd8b2fb9e1039e38e6, State: Initialized, Role: FOLLOWER
I20260812 01:03:48.200819 25498 consensus_queue.cc:260] T 33fbc39f37314f90906e710d7bdc5abc P 0031233573a748cd8b2fb9e1039e38e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1093ae1b58084a85b5ff97511c67c2f0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 44735 } } peers { permanent_uuid: "0031233573a748cd8b2fb9e1039e38e6" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 46573 } } peers { permanent_uuid: "e9bdf277f8084e8e9a982268f5fb245a" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35453 } }
I20260812 01:03:48.201793 25499 raft_consensus.cc:348] T 33fbc39f37314f90906e710d7bdc5abc P 1093ae1b58084a85b5ff97511c67c2f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1093ae1b58084a85b5ff97511c67c2f0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 44735 } } peers { permanent_uuid: "0031233573a748cd8b2fb9e1039e38e6" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 46573 } } peers { permanent_uuid: "e9bdf277f8084e8e9a982268f5fb245a" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35453 } }
I20260812 01:03:48.202353 25499 raft_consensus.cc:374] T 33fbc39f37314f90906e710d7bdc5abc P 1093ae1b58084a85b5ff97511c67c2f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:48.202476 25499 raft_consensus.cc:729] T 33fbc39f37314f90906e710d7bdc5abc P 1093ae1b58084a85b5ff97511c67c2f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1093ae1b58084a85b5ff97511c67c2f0, State: Initialized, Role: FOLLOWER
I20260812 01:03:48.203181 25499 consensus_queue.cc:260] T 33fbc39f37314f90906e710d7bdc5abc P 1093ae1b58084a85b5ff97511c67c2f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1093ae1b58084a85b5ff97511c67c2f0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 44735 } } peers { permanent_uuid: "0031233573a748cd8b2fb9e1039e38e6" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 46573 } } peers { permanent_uuid: "e9bdf277f8084e8e9a982268f5fb245a" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35453 } }
I20260812 01:03:48.205353 25498 ts_tablet_manager.cc:1428] T 33fbc39f37314f90906e710d7bdc5abc P 0031233573a748cd8b2fb9e1039e38e6: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260812 01:03:48.205397 25456 heartbeater.cc:503] Master 127.21.201.126:36781 was elected leader, sending a full tablet report...
I20260812 01:03:48.207063 25226 heartbeater.cc:503] Master 127.21.201.126:36781 was elected leader, sending a full tablet report...
I20260812 01:03:48.207672 25499 ts_tablet_manager.cc:1428] T 33fbc39f37314f90906e710d7bdc5abc P 1093ae1b58084a85b5ff97511c67c2f0: Time spent starting tablet: real 0.018s user 0.011s sys 0.005s
W20260812 01:03:48.337855 25458 tablet.cc:2369] T 33fbc39f37314f90906e710d7bdc5abc P 0031233573a748cd8b2fb9e1039e38e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:03:48.375445 25229 tablet.cc:2369] T 33fbc39f37314f90906e710d7bdc5abc P 1093ae1b58084a85b5ff97511c67c2f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 01:03:48.393594 24984 tablet.cc:2369] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:03:48.549717 25506 raft_consensus.cc:482] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 01:03:48.550138 25506 raft_consensus.cc:504] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1093ae1b58084a85b5ff97511c67c2f0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 44735 } } peers { permanent_uuid: "0031233573a748cd8b2fb9e1039e38e6" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 46573 } } peers { permanent_uuid: "e9bdf277f8084e8e9a982268f5fb245a" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35453 } }
I20260812 01:03:48.551959 25506 leader_election.cc:288] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1093ae1b58084a85b5ff97511c67c2f0 (127.21.201.66:44735), 0031233573a748cd8b2fb9e1039e38e6 (127.21.201.67:46573)
I20260812 01:03:48.558890 25487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:48.552548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:56597 (local address 127.21.201.67:46573)
0812 01:03:48.552791 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:48.552797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:48.553445 (+ 648us) server_negotiation.cc:407] Connection header received
0812 01:03:48.553611 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:48.553616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:48.553679 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:48.553792 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:48.554498 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.555254 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.557197 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.557957 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.558114 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:48.558196 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:48.558496 (+ 300us) server_negotiation.cc:299] Negotiation successful
0812 01:03:48.558629 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 01:03:48.558943 25488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:48.552304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.201.65:49719 (local address 127.21.201.66:44735)
0812 01:03:48.552547 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:48.552554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:48.553307 (+ 753us) server_negotiation.cc:407] Connection header received
0812 01:03:48.553538 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:48.553546 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:48.553644 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:48.553715 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 01:03:48.554386 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.555122 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.557251 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.558002 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.558190 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:48.558260 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:48.558546 (+ 286us) server_negotiation.cc:299] Negotiation successful
0812 01:03:48.558673 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 01:03:48.559001 25512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:48.552141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.66:44735 (local address 127.21.201.65:49719)
0812 01:03:48.553162 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:48.553187 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:48.553350 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:48.553760 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:48.553768 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:48.553794 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:48.554221 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:48.554234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.555254 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.555260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:48.557020 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:48.557032 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.558225 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.558232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:48.558370 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:48.558391 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 01:03:48.558498 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 01:03:48.558619 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"mutex_wait_us":94,"thread_start_us":616,"threads_started":1}
I20260812 01:03:48.559253 25513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:48.552501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.67:46573 (local address 127.21.201.65:56597)
0812 01:03:48.553315 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:48.553334 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:03:48.553482 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:03:48.553830 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:03:48.553842 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:03:48.553874 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 01:03:48.554329 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:48.554344 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.555429 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.555432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:48.557004 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:03:48.557017 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.558148 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.558154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:03:48.558333 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:48.558367 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 01:03:48.558441 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 01:03:48.558584 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"mutex_wait_us":21,"thread_start_us":477,"threads_started":1}
I20260812 01:03:48.560081 25149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33fbc39f37314f90906e710d7bdc5abc" candidate_uuid: "e9bdf277f8084e8e9a982268f5fb245a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1093ae1b58084a85b5ff97511c67c2f0" is_pre_election: true
I20260812 01:03:48.560160 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33fbc39f37314f90906e710d7bdc5abc" candidate_uuid: "e9bdf277f8084e8e9a982268f5fb245a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0031233573a748cd8b2fb9e1039e38e6" is_pre_election: true
I20260812 01:03:48.560635 25369 raft_consensus.cc:2454] T 33fbc39f37314f90906e710d7bdc5abc P 0031233573a748cd8b2fb9e1039e38e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9bdf277f8084e8e9a982268f5fb245a in term 0.
I20260812 01:03:48.560621 25149 raft_consensus.cc:2454] T 33fbc39f37314f90906e710d7bdc5abc P 1093ae1b58084a85b5ff97511c67c2f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9bdf277f8084e8e9a982268f5fb245a in term 0.
I20260812 01:03:48.561233 24785 leader_election.cc:302] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1093ae1b58084a85b5ff97511c67c2f0, e9bdf277f8084e8e9a982268f5fb245a; no voters:
I20260812 01:03:48.561641 25506 raft_consensus.cc:2781] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 01:03:48.561805 25506 raft_consensus.cc:482] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 01:03:48.561934 25506 raft_consensus.cc:3037] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:48.564263 25506 raft_consensus.cc:504] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1093ae1b58084a85b5ff97511c67c2f0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 44735 } } peers { permanent_uuid: "0031233573a748cd8b2fb9e1039e38e6" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 46573 } } peers { permanent_uuid: "e9bdf277f8084e8e9a982268f5fb245a" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35453 } }
I20260812 01:03:48.564941 25506 leader_election.cc:288] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a [CANDIDATE]: Term 1 election: Requested vote from peers 1093ae1b58084a85b5ff97511c67c2f0 (127.21.201.66:44735), 0031233573a748cd8b2fb9e1039e38e6 (127.21.201.67:46573)
I20260812 01:03:48.565272 25149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33fbc39f37314f90906e710d7bdc5abc" candidate_uuid: "e9bdf277f8084e8e9a982268f5fb245a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1093ae1b58084a85b5ff97511c67c2f0"
I20260812 01:03:48.565488 25149 raft_consensus.cc:3037] T 33fbc39f37314f90906e710d7bdc5abc P 1093ae1b58084a85b5ff97511c67c2f0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:48.565575 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33fbc39f37314f90906e710d7bdc5abc" candidate_uuid: "e9bdf277f8084e8e9a982268f5fb245a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0031233573a748cd8b2fb9e1039e38e6"
I20260812 01:03:48.565798 25369 raft_consensus.cc:3037] T 33fbc39f37314f90906e710d7bdc5abc P 0031233573a748cd8b2fb9e1039e38e6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:48.568915 25149 raft_consensus.cc:2454] T 33fbc39f37314f90906e710d7bdc5abc P 1093ae1b58084a85b5ff97511c67c2f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9bdf277f8084e8e9a982268f5fb245a in term 1.
I20260812 01:03:48.569165 25369 raft_consensus.cc:2454] T 33fbc39f37314f90906e710d7bdc5abc P 0031233573a748cd8b2fb9e1039e38e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9bdf277f8084e8e9a982268f5fb245a in term 1.
I20260812 01:03:48.569320 24785 leader_election.cc:302] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1093ae1b58084a85b5ff97511c67c2f0, e9bdf277f8084e8e9a982268f5fb245a; no voters:
I20260812 01:03:48.569681 25506 raft_consensus.cc:2781] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:03:48.570164 25506 raft_consensus.cc:686] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a [term 1 LEADER]: Becoming Leader. State: Replica: e9bdf277f8084e8e9a982268f5fb245a, State: Running, Role: LEADER
I20260812 01:03:48.570605 25506 consensus_queue.cc:237] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1093ae1b58084a85b5ff97511c67c2f0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 44735 } } peers { permanent_uuid: "0031233573a748cd8b2fb9e1039e38e6" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 46573 } } peers { permanent_uuid: "e9bdf277f8084e8e9a982268f5fb245a" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35453 } }
I20260812 01:03:48.575240 24655 catalog_manager.cc:5495] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a reported cstate change: term changed from 0 to 1, leader changed from <none> to e9bdf277f8084e8e9a982268f5fb245a (127.21.201.65). New cstate: current_term: 1 leader_uuid: "e9bdf277f8084e8e9a982268f5fb245a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1093ae1b58084a85b5ff97511c67c2f0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 44735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0031233573a748cd8b2fb9e1039e38e6" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 46573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9bdf277f8084e8e9a982268f5fb245a" member_type: VOTER last_known_addr { host: "127.21.201.65" port: 35453 } health_report { overall_health: HEALTHY } } }
I20260812 01:03:48.615645 25494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:48.610252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.21.201.65:35453)
0812 01:03:48.610480 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:48.610486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:48.610508 (+ 22us) server_negotiation.cc:407] Connection header received
0812 01:03:48.610737 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:48.610742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:48.610839 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:48.611001 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:03:48.611349 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.612129 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.613500 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.613925 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.614060 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:48.614533 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:03:48.615053 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:03:48.615216 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:48.615283 (+ 67us) server_negotiation.cc:299] Negotiation successful
0812 01:03:48.615376 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 01:03:48.624498 25149 raft_consensus.cc:1261] T 33fbc39f37314f90906e710d7bdc5abc P 1093ae1b58084a85b5ff97511c67c2f0 [term 1 FOLLOWER]: Refusing update from remote peer e9bdf277f8084e8e9a982268f5fb245a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:03:48.624562 25369 raft_consensus.cc:1261] T 33fbc39f37314f90906e710d7bdc5abc P 0031233573a748cd8b2fb9e1039e38e6 [term 1 FOLLOWER]: Refusing update from remote peer e9bdf277f8084e8e9a982268f5fb245a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:03:48.625561 25515 consensus_queue.cc:1035] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1093ae1b58084a85b5ff97511c67c2f0" member_type: VOTER last_known_addr { host: "127.21.201.66" port: 44735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:03:48.625986 25506 consensus_queue.cc:1035] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0031233573a748cd8b2fb9e1039e38e6" member_type: VOTER last_known_addr { host: "127.21.201.67" port: 46573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:03:48.637300 25521 mvcc.cc:205] Tried to move back new op lower bound from 7317490190835105792 to 7317490190628593664. Current Snapshot: MvccSnapshot[applied={T|T < 7317490190835105792}]
I20260812 01:03:48.657691 24655 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:03:48.657985 24655 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 01:03:48.661477 24655 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04f18efa67194fcfa1fb65462c0c0c1e: Sending DeleteTablet for 3 replicas of tablet 33fbc39f37314f90906e710d7bdc5abc
I20260812 01:03:48.662637 25102 tablet_service.cc:1514] Processing DeleteTablet for tablet 33fbc39f37314f90906e710d7bdc5abc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:03:48 UTC) from {username='slave'} at 127.0.0.1:43442
I20260812 01:03:48.662639 25339 tablet_service.cc:1514] Processing DeleteTablet for tablet 33fbc39f37314f90906e710d7bdc5abc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:03:48 UTC) from {username='slave'} at 127.0.0.1:54184
I20260812 01:03:48.662885 24853 tablet_service.cc:1514] Processing DeleteTablet for tablet 33fbc39f37314f90906e710d7bdc5abc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 01:03:48 UTC) from {username='slave'} at 127.0.0.1:45000
I20260812 01:03:48.664907 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 24750
I20260812 01:03:48.681006 25537 tablet_replica.cc:330] T 33fbc39f37314f90906e710d7bdc5abc P 1093ae1b58084a85b5ff97511c67c2f0: stopping tablet replica
I20260812 01:03:48.686283 25537 raft_consensus.cc:2229] T 33fbc39f37314f90906e710d7bdc5abc P 1093ae1b58084a85b5ff97511c67c2f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:03:48.687130 25537 raft_consensus.cc:2258] T 33fbc39f37314f90906e710d7bdc5abc P 1093ae1b58084a85b5ff97511c67c2f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:03:48.689617 25534 tablet_replica.cc:330] T 33fbc39f37314f90906e710d7bdc5abc P 0031233573a748cd8b2fb9e1039e38e6: stopping tablet replica
I20260812 01:03:48.690181 25534 raft_consensus.cc:2229] T 33fbc39f37314f90906e710d7bdc5abc P 0031233573a748cd8b2fb9e1039e38e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 01:03:48.690778 25534 raft_consensus.cc:2258] T 33fbc39f37314f90906e710d7bdc5abc P 0031233573a748cd8b2fb9e1039e38e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:03:48.693481 25534 ts_tablet_manager.cc:1905] T 33fbc39f37314f90906e710d7bdc5abc P 0031233573a748cd8b2fb9e1039e38e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:03:48.693835 25537 ts_tablet_manager.cc:1905] T 33fbc39f37314f90906e710d7bdc5abc P 1093ae1b58084a85b5ff97511c67c2f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 01:03:48.696712 25494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:48.666793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45024 (local address 127.21.201.65:35453)
0812 01:03:48.667040 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:48.667049 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:48.678600 (+ 11551us) server_negotiation.cc:407] Connection header received
0812 01:03:48.679206 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:48.679216 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:48.679395 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:48.679495 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:48.681335 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.683017 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.687856 (+ 4839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.688832 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.688980 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:48.694876 (+ 5896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:48.694887 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:48.694904 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:48.695708 (+ 804us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:48.695799 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:48.695827 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:48.695929 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:48.696165 (+ 236us) server_negotiation.cc:299] Negotiation successful
0812 01:03:48.696285 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 01:03:48.697235 25538 tablet_replica.cc:330] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a: stopping tablet replica
I20260812 01:03:48.697713 25538 raft_consensus.cc:2229] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a [term 1 LEADER]: Raft consensus shutting down.
I20260812 01:03:48.697954 24971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:48.703220 25534 ts_tablet_manager.cc:1918] T 33fbc39f37314f90906e710d7bdc5abc P 0031233573a748cd8b2fb9e1039e38e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:03:48.703421 25534 log.cc:1196] T 33fbc39f37314f90906e710d7bdc5abc P 0031233573a748cd8b2fb9e1039e38e6: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-2/wal/wals/33fbc39f37314f90906e710d7bdc5abc
I20260812 01:03:48.703424 25537 ts_tablet_manager.cc:1918] T 33fbc39f37314f90906e710d7bdc5abc P 1093ae1b58084a85b5ff97511c67c2f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 01:03:48.703577 25537 log.cc:1196] T 33fbc39f37314f90906e710d7bdc5abc P 1093ae1b58084a85b5ff97511c67c2f0: Deleting WAL directory at /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786496278741276-22309-0/minicluster-data/ts-1/wal/wals/33fbc39f37314f90906e710d7bdc5abc
I20260812 01:03:48.704119 25534 ts_tablet_manager.cc:1939] T 33fbc39f37314f90906e710d7bdc5abc P 0031233573a748cd8b2fb9e1039e38e6: Deleting consensus metadata
I20260812 01:03:48.704149 25537 ts_tablet_manager.cc:1939] T 33fbc39f37314f90906e710d7bdc5abc P 1093ae1b58084a85b5ff97511c67c2f0: Deleting consensus metadata
I20260812 01:03:48.706090 24636 catalog_manager.cc:4841] TS 1093ae1b58084a85b5ff97511c67c2f0 (127.21.201.66:44735): tablet 33fbc39f37314f90906e710d7bdc5abc (table test-table [id=e63e3d47fc2444d792082b44a52d76a1]) successfully deleted
I20260812 01:03:48.706100 24634 catalog_manager.cc:4841] TS 0031233573a748cd8b2fb9e1039e38e6 (127.21.201.67:46573): tablet 33fbc39f37314f90906e710d7bdc5abc (table test-table [id=e63e3d47fc2444d792082b44a52d76a1]) successfully deleted
I20260812 01:03:48.797619 25538 raft_consensus.cc:2258] T 33fbc39f37314f90906e710d7bdc5abc P e9bdf277f8084e8e9a982268f5fb245a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 01:03:48.798177 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 24750
W20260812 01:03:48.824868 24635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.201.65:35453 (error 108)
I20260812 01:03:48.825171 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 24988
I20260812 01:03:48.826989 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:48.826193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:35453 (local address 127.0.0.1:45034)
0812 01:03:48.826674 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:48.826800 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:35453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":247,"threads_started":1}
W20260812 01:03:48.827240 24635 catalog_manager.cc:4584] TS e9bdf277f8084e8e9a982268f5fb245a (127.21.201.65:35453): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 33fbc39f37314f90906e710d7bdc5abc: Network error: Client connection negotiation failed: client connection to 127.21.201.65:35453: connect: Connection refused (error 111)
I20260812 01:03:48.832374 25488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:48.826094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43456 (local address 127.21.201.66:44735)
0812 01:03:48.826336 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:48.826342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:48.826364 (+ 22us) server_negotiation.cc:407] Connection header received
0812 01:03:48.826465 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:48.826481 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:48.826573 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:48.826698 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:48.827112 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.828027 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.829306 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.829741 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.829884 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:48.830940 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:48.830948 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:48.830962 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:48.831557 (+ 595us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:48.831605 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:48.831626 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:48.831793 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:48.831932 (+ 139us) server_negotiation.cc:299] Negotiation successful
0812 01:03:48.832041 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 01:03:48.833045 25215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:48.880954 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:48.880458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:35453 (local address 127.0.0.1:45040)
0812 01:03:48.880687 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:48.880786 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:35453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260812 01:03:48.882087 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:48.881727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:35453 (local address 127.0.0.1:45044)
0812 01:03:48.881866 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:48.881932 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:35453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260812 01:03:48.925316 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 24988
I20260812 01:03:48.942736 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:48.942249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:35453 (local address 127.0.0.1:45046)
0812 01:03:48.942470 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:48.942570 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:35453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260812 01:03:48.943842 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:48.943412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:35453 (local address 127.0.0.1:45062)
0812 01:03:48.943557 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:48.943630 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:35453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260812 01:03:48.953277 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 25232
I20260812 01:03:48.960466 25487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:48.954076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54194 (local address 127.21.201.67:46573)
0812 01:03:48.954297 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:48.954303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:48.954325 (+ 22us) server_negotiation.cc:407] Connection header received
0812 01:03:48.954534 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:48.954539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:48.954660 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:48.954775 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:48.955194 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.956010 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.957249 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:48.957684 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:48.957822 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:48.958961 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:48.958970 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:48.958998 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:48.959571 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:48.959619 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:48.959639 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:48.959870 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:48.960011 (+ 141us) server_negotiation.cc:299] Negotiation successful
0812 01:03:48.960132 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 01:03:48.961158 25446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:49.024648 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:49.024113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:35453 (local address 127.0.0.1:45074)
0812 01:03:49.024340 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:49.024452 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:35453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260812 01:03:49.025871 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:49.025395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:35453 (local address 127.0.0.1:45090)
0812 01:03:49.025625 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:49.025710 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:35453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260812 01:03:49.052340 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 25232
I20260812 01:03:49.084331 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 24601
I20260812 01:03:49.085489 24695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:49.198974 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:49.198283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.65:35453 (local address 127.0.0.1:45092)
0812 01:03:49.198640 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0812 01:03:49.198751 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.65:35453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189}
I20260812 01:03:49.198928 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 24601
2026-08-12T01:03:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3216 ms)
[----------] 6 tests from EncryptionPolicyTest (25088 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:03:49 dist-test-slave-7f01 krb5kdc[25552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:03:49 dist-test-slave-7f01 krb5kdc[25552](info): set up 2 sockets
Aug 12 01:03:49 dist-test-slave-7f01 krb5kdc[25552](info): commencing operation
krb5kdc: starting...
W20260812 01:03:50.590613 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.339s user 0.004s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:03:50 dist-test-slave-7f01 krb5kdc[25552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:03:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:03:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 01:03:50.916182 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41885
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:03:51.272528 25570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:51.272850 25570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:51.272958 25570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:51.280304 25570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:03:51.280421 25570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:51.280469 25570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:51.280534 25570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:03:51.280579 25570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:03:51.292126 25570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41885
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:51.293442 25570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:51.294714 25570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:51.306497 25578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:51.306661 25581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:51.306535 25579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:51.307663 25570 server_base.cc:1034] running on GCE node
I20260812 01:03:51.308569 25570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:51.310014 25570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:51.311246 25570 hybrid_clock.cc:648] HybridClock initialized: now 1786496631311209 us; error 56 us; skew 500 ppm
I20260812 01:03:51.311606 25570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:03:51 dist-test-slave-7f01 krb5kdc[25552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:03:51.322089 25570 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 01:03:51.323781 25570 webserver.cc:460] Webserver started at http://127.0.0.1:46591/ using document root <none> and password file <none>
I20260812 01:03:51.324507 25570 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:51.324653 25570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:51.324963 25570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:51.328296 25570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "c99a38c11e1e4dd7a9ced04b0a693e4b"
format_stamp: "Formatted at 2026-08-12 01:03:51 on dist-test-slave-7f01"
server_key: "be9c08ac75ef884a82cf2de81d5981ad"
server_key_iv: "4fe9c65a2d630097c7ad6606f88d41fc"
server_key_version: "encryptionkey@0"
I20260812 01:03:51.329094 25570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "c99a38c11e1e4dd7a9ced04b0a693e4b"
format_stamp: "Formatted at 2026-08-12 01:03:51 on dist-test-slave-7f01"
server_key: "be9c08ac75ef884a82cf2de81d5981ad"
server_key_iv: "4fe9c65a2d630097c7ad6606f88d41fc"
server_key_version: "encryptionkey@0"
I20260812 01:03:51.334300 25570 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:03:51.337878 25593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:51.339195 25570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 01:03:51.339385 25570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "c99a38c11e1e4dd7a9ced04b0a693e4b"
format_stamp: "Formatted at 2026-08-12 01:03:51 on dist-test-slave-7f01"
server_key: "be9c08ac75ef884a82cf2de81d5981ad"
server_key_iv: "4fe9c65a2d630097c7ad6606f88d41fc"
server_key_version: "encryptionkey@0"
I20260812 01:03:51.339596 25570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:51.364872 25570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:51.369239 25570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:51.369549 25570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:51.389072 25570 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41885
I20260812 01:03:51.389117 25692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41885 every 8 connection(s)
I20260812 01:03:51.390483 25570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:03:51.394246 25694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:03:51.398094 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 25570
I20260812 01:03:51.398346 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:03:51.399184 22309 external_mini_cluster.cc:1428] Setting key 94b622865fc5a260a8e507c23773ab87
I20260812 01:03:51.408133 25694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99a38c11e1e4dd7a9ced04b0a693e4b: Bootstrap starting.
I20260812 01:03:51.412156 25694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c99a38c11e1e4dd7a9ced04b0a693e4b: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:51.413580 25694 log.cc:824] T 00000000000000000000000000000000 P c99a38c11e1e4dd7a9ced04b0a693e4b: Log is configured to *not* fsync() on all Append() calls
Aug 12 01:03:51 dist-test-slave-7f01 krb5kdc[25552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 01:03:51.417022 25694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99a38c11e1e4dd7a9ced04b0a693e4b: No bootstrap required, opened a new log
I20260812 01:03:51.424302 25697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:51.401270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60845 (local address 127.0.0.1:41885)
0812 01:03:51.402045 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:51.402057 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:51.402099 (+ 42us) server_negotiation.cc:407] Connection header received
0812 01:03:51.403806 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:51.403828 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:51.404214 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:51.404544 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:51.406475 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:51.407895 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:51.409690 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:51.410379 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:51.410502 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:51.415972 (+ 5470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:51.415980 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:51.415994 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:51.416034 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:51.419490 (+ 3456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:51.420538 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:51.420545 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:51.420551 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:51.420743 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:51.421272 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:51.421276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:51.421278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:51.421855 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:51.422142 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:51.422727 (+ 585us) server_negotiation.cc:299] Negotiation successful
0812 01:03:51.423083 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":364,"threads_started":1}
I20260812 01:03:51.427241 25694 raft_consensus.cc:348] T 00000000000000000000000000000000 P c99a38c11e1e4dd7a9ced04b0a693e4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99a38c11e1e4dd7a9ced04b0a693e4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41885 } }
I20260812 01:03:51.427626 25694 raft_consensus.cc:374] T 00000000000000000000000000000000 P c99a38c11e1e4dd7a9ced04b0a693e4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:51.427784 25694 raft_consensus.cc:729] T 00000000000000000000000000000000 P c99a38c11e1e4dd7a9ced04b0a693e4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c99a38c11e1e4dd7a9ced04b0a693e4b, State: Initialized, Role: FOLLOWER
I20260812 01:03:51.428448 25694 consensus_queue.cc:260] T 00000000000000000000000000000000 P c99a38c11e1e4dd7a9ced04b0a693e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99a38c11e1e4dd7a9ced04b0a693e4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41885 } }
I20260812 01:03:51.428622 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:03:51.428723 25694 raft_consensus.cc:388] T 00000000000000000000000000000000 P c99a38c11e1e4dd7a9ced04b0a693e4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:03:51.428862 25694 raft_consensus.cc:482] T 00000000000000000000000000000000 P c99a38c11e1e4dd7a9ced04b0a693e4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:03:51.429039 25694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c99a38c11e1e4dd7a9ced04b0a693e4b [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:51.431329 25694 raft_consensus.cc:504] T 00000000000000000000000000000000 P c99a38c11e1e4dd7a9ced04b0a693e4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99a38c11e1e4dd7a9ced04b0a693e4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41885 } }
I20260812 01:03:51.431943 25694 leader_election.cc:302] T 00000000000000000000000000000000 P c99a38c11e1e4dd7a9ced04b0a693e4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c99a38c11e1e4dd7a9ced04b0a693e4b; no voters:
I20260812 01:03:51.432627 25694 leader_election.cc:288] T 00000000000000000000000000000000 P c99a38c11e1e4dd7a9ced04b0a693e4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:03:51.432729 25706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c99a38c11e1e4dd7a9ced04b0a693e4b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:03:51.435034 25706 raft_consensus.cc:686] T 00000000000000000000000000000000 P c99a38c11e1e4dd7a9ced04b0a693e4b [term 1 LEADER]: Becoming Leader. State: Replica: c99a38c11e1e4dd7a9ced04b0a693e4b, State: Running, Role: LEADER
I20260812 01:03:51.435206 25694 sys_catalog.cc:564] T 00000000000000000000000000000000 P c99a38c11e1e4dd7a9ced04b0a693e4b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:03:51.435602 25706 consensus_queue.cc:237] T 00000000000000000000000000000000 P c99a38c11e1e4dd7a9ced04b0a693e4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99a38c11e1e4dd7a9ced04b0a693e4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41885 } }
I20260812 01:03:51.438848 25710 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99a38c11e1e4dd7a9ced04b0a693e4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c99a38c11e1e4dd7a9ced04b0a693e4b. Latest consensus state: current_term: 1 leader_uuid: "c99a38c11e1e4dd7a9ced04b0a693e4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99a38c11e1e4dd7a9ced04b0a693e4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41885 } } }
I20260812 01:03:51.439281 25710 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99a38c11e1e4dd7a9ced04b0a693e4b [sys.catalog]: This master's current role is: LEADER
I20260812 01:03:51.452634 25709 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99a38c11e1e4dd7a9ced04b0a693e4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c99a38c11e1e4dd7a9ced04b0a693e4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99a38c11e1e4dd7a9ced04b0a693e4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41885 } } }
I20260812 01:03:51.453399 25709 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99a38c11e1e4dd7a9ced04b0a693e4b [sys.catalog]: This master's current role is: LEADER
I20260812 01:03:51.453483 25723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:03:51.454582 25697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:51.434338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53695 (local address 127.0.0.1:41885)
0812 01:03:51.434536 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:51.434542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:51.434566 (+ 24us) server_negotiation.cc:407] Connection header received
0812 01:03:51.434627 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:51.434633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:51.434713 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:51.434835 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:51.436641 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:51.437532 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:51.439595 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:51.440562 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:51.440728 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:51.445335 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:51.445345 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:51.445353 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:51.445409 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:51.450317 (+ 4908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:51.451236 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:51.451242 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:51.451245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:51.451392 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:51.452077 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:51.452082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:51.452084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:51.452617 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:51.452972 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:51.454168 (+ 1196us) server_negotiation.cc:299] Negotiation successful
0812 01:03:51.454317 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 01:03:51.461180 25723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:03:51.471222 25723 catalog_manager.cc:1347] Generated new cluster ID: e7c11b0feb3141988768dd9531fbe034
I20260812 01:03:51.471370 25723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:03:51.497277 25723 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:03:51.498670 25723 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:03:51.505019 25723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c99a38c11e1e4dd7a9ced04b0a693e4b: Generated new TSK 0
I20260812 01:03:51.505808 25723 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:03:51.517640 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 25570
I20260812 01:03:51.519102 25683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:51.614514 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 25570
W20260812 01:03:51.645143 25550 connection.cc:535] client connection to 127.0.0.1:41885 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41885): Connection reset by peer (error 104)
2026-08-12T01:03:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:03:51 dist-test-slave-7f01 krb5kdc[25742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:03:51 dist-test-slave-7f01 krb5kdc[25742](info): set up 2 sockets
Aug 12 01:03:51 dist-test-slave-7f01 krb5kdc[25742](info): commencing operation
krb5kdc: starting...
W20260812 01:03:53.006520 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.342s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:03:53 dist-test-slave-7f01 krb5kdc[25742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:03:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:03:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 01:03:53.319583 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35409
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:03:53.670498 25763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:53.670775 25763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:53.670960 25763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:53.678069 25763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:03:53.678174 25763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:53.678300 25763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:53.678409 25763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:03:53.678496 25763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:03:53.688061 25763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35409
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:53.689291 25763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:53.690618 25763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:53.702656 25772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:53.702744 25770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:53.702764 25775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:53.703596 25763 server_base.cc:1034] running on GCE node
I20260812 01:03:53.704416 25763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:53.705687 25763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:53.706905 25763 hybrid_clock.cc:648] HybridClock initialized: now 1786496633706851 us; error 43 us; skew 500 ppm
I20260812 01:03:53.707267 25763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:03:53 dist-test-slave-7f01 krb5kdc[25742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:03:53.710729 25763 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 01:03:53.712329 25763 webserver.cc:460] Webserver started at http://127.0.0.1:44711/ using document root <none> and password file <none>
I20260812 01:03:53.712970 25763 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:53.713076 25763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:53.713327 25763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:53.716248 25763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "4b660be3b5bf460280c9bf22aa09f83e"
format_stamp: "Formatted at 2026-08-12 01:03:53 on dist-test-slave-7f01"
server_key: "52310a6b4591df9d224361a34a53c572"
server_key_iv: "972148757d23b1ac794023bbcfbc83b0"
server_key_version: "encryptionkey@0"
I20260812 01:03:53.716987 25763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "4b660be3b5bf460280c9bf22aa09f83e"
format_stamp: "Formatted at 2026-08-12 01:03:53 on dist-test-slave-7f01"
server_key: "52310a6b4591df9d224361a34a53c572"
server_key_iv: "972148757d23b1ac794023bbcfbc83b0"
server_key_version: "encryptionkey@0"
I20260812 01:03:53.722175 25763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260812 01:03:53.725646 25791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:53.726976 25763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260812 01:03:53.727167 25763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "4b660be3b5bf460280c9bf22aa09f83e"
format_stamp: "Formatted at 2026-08-12 01:03:53 on dist-test-slave-7f01"
server_key: "52310a6b4591df9d224361a34a53c572"
server_key_iv: "972148757d23b1ac794023bbcfbc83b0"
server_key_version: "encryptionkey@0"
I20260812 01:03:53.727370 25763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:53.742411 25763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:53.746845 25763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:53.747166 25763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:53.766469 25763 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35409
I20260812 01:03:53.766494 25893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35409 every 8 connection(s)
I20260812 01:03:53.767952 25763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:03:53.770602 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 25763
I20260812 01:03:53.770797 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:03:53.771384 22309 external_mini_cluster.cc:1428] Setting key 781b20416fbbf5b708694b896079ef58
I20260812 01:03:53.772644 25894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:03:53.785127 25897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:53.773259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46433 (local address 127.0.0.1:35409)
0812 01:03:53.774058 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:53.774069 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:53.774109 (+ 40us) server_negotiation.cc:407] Connection header received
0812 01:03:53.775709 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:53.775729 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:53.776190 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:53.776503 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:53.777127 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:53.778464 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:53.780456 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:53.781138 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:53.781263 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:53.782674 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:53.782685 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:53.782697 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:53.783286 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:53.783370 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:53.783398 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:53.783535 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:53.783993 (+ 458us) server_negotiation.cc:299] Negotiation successful
0812 01:03:53.784179 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":378,"threads_started":1}
I20260812 01:03:53.786264 25894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b660be3b5bf460280c9bf22aa09f83e: Bootstrap starting.
I20260812 01:03:53.788967 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:03:53.789706 25894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b660be3b5bf460280c9bf22aa09f83e: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:53.790936 25894 log.cc:824] T 00000000000000000000000000000000 P 4b660be3b5bf460280c9bf22aa09f83e: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:53.794430 25894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b660be3b5bf460280c9bf22aa09f83e: No bootstrap required, opened a new log
I20260812 01:03:53.801039 25897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:53.794665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43853 (local address 127.0.0.1:35409)
0812 01:03:53.794820 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:53.794824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:53.795283 (+ 459us) server_negotiation.cc:407] Connection header received
0812 01:03:53.795496 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:53.795501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:53.795570 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:53.795676 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:53.796246 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:53.797081 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:53.798367 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:53.798890 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:53.799057 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:53.800276 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:53.800285 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:53.800288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:53.800304 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:53.800345 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:53.800360 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:53.800482 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:53.800684 (+ 202us) server_negotiation.cc:299] Negotiation successful
0812 01:03:53.800743 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 01:03:53.803118 25894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b660be3b5bf460280c9bf22aa09f83e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b660be3b5bf460280c9bf22aa09f83e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35409 } }
I20260812 01:03:53.803540 25894 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b660be3b5bf460280c9bf22aa09f83e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:53.803678 25894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b660be3b5bf460280c9bf22aa09f83e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b660be3b5bf460280c9bf22aa09f83e, State: Initialized, Role: FOLLOWER
I20260812 01:03:53.804375 25894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b660be3b5bf460280c9bf22aa09f83e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b660be3b5bf460280c9bf22aa09f83e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35409 } }
I20260812 01:03:53.804627 25894 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b660be3b5bf460280c9bf22aa09f83e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:03:53.804769 25894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b660be3b5bf460280c9bf22aa09f83e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:03:53.804962 25894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b660be3b5bf460280c9bf22aa09f83e [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:53.807797 25894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b660be3b5bf460280c9bf22aa09f83e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b660be3b5bf460280c9bf22aa09f83e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35409 } }
I20260812 01:03:53.808336 25894 leader_election.cc:302] T 00000000000000000000000000000000 P 4b660be3b5bf460280c9bf22aa09f83e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b660be3b5bf460280c9bf22aa09f83e; no voters:
I20260812 01:03:53.808972 25894 leader_election.cc:288] T 00000000000000000000000000000000 P 4b660be3b5bf460280c9bf22aa09f83e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:03:53.809128 25907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b660be3b5bf460280c9bf22aa09f83e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:03:53.809880 25907 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b660be3b5bf460280c9bf22aa09f83e [term 1 LEADER]: Becoming Leader. State: Replica: 4b660be3b5bf460280c9bf22aa09f83e, State: Running, Role: LEADER
I20260812 01:03:53.810425 25907 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b660be3b5bf460280c9bf22aa09f83e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b660be3b5bf460280c9bf22aa09f83e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35409 } }
I20260812 01:03:53.811179 25894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b660be3b5bf460280c9bf22aa09f83e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:03:53.813112 25909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b660be3b5bf460280c9bf22aa09f83e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b660be3b5bf460280c9bf22aa09f83e. Latest consensus state: current_term: 1 leader_uuid: "4b660be3b5bf460280c9bf22aa09f83e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b660be3b5bf460280c9bf22aa09f83e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35409 } } }
I20260812 01:03:53.813501 25909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b660be3b5bf460280c9bf22aa09f83e [sys.catalog]: This master's current role is: LEADER
I20260812 01:03:53.815215 25913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:03:53.815165 25908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b660be3b5bf460280c9bf22aa09f83e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b660be3b5bf460280c9bf22aa09f83e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b660be3b5bf460280c9bf22aa09f83e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35409 } } }
I20260812 01:03:53.815508 25908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b660be3b5bf460280c9bf22aa09f83e [sys.catalog]: This master's current role is: LEADER
I20260812 01:03:53.822105 25913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:03:53.830657 25913 catalog_manager.cc:1347] Generated new cluster ID: 9b1b929301144f09b4fa5fbb1a5d81b6
I20260812 01:03:53.830777 25913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:03:53.842430 25913 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:03:53.843983 25913 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:03:53.853993 25913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b660be3b5bf460280c9bf22aa09f83e: Generated new TSK 0
I20260812 01:03:53.854786 25913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:03:53.865238 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 25763
W20260812 01:03:53.866611 25883 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46433
W20260812 01:03:53.867471 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 25763: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:03:53.867677 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 25763
W20260812 01:03:53.902381 25740 connection.cc:535] client connection to 127.0.0.1:35409 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35409): Connection reset by peer (error 104)
2026-08-12T01:03:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2256 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:03:53 dist-test-slave-7f01 krb5kdc[25940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:03:53 dist-test-slave-7f01 krb5kdc[25940](info): set up 2 sockets
Aug 12 01:03:53 dist-test-slave-7f01 krb5kdc[25940](info): commencing operation
krb5kdc: starting...
W20260812 01:03:55.262537 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.341s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:03:55 dist-test-slave-7f01 krb5kdc[25940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:03:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:03:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 01:03:55.585300 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40391
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:03:55.947921 25969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:55.948220 25969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:55.948437 25969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:55.956012 25969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:03:55.956130 25969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:55.956264 25969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:55.956377 25969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:03:55.956462 25969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:03:55.966463 25969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40391
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:55.967814 25969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:55.969162 25969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:55.981746 25978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:55.981843 25979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:55.981784 25988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:55.982600 25969 server_base.cc:1034] running on GCE node
I20260812 01:03:55.983449 25969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:55.984997 25969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:55.986303 25969 hybrid_clock.cc:648] HybridClock initialized: now 1786496635986261 us; error 50 us; skew 500 ppm
I20260812 01:03:55.986712 25969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:03:55 dist-test-slave-7f01 krb5kdc[25940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:03:55.990072 25969 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 01:03:55.991622 25969 webserver.cc:460] Webserver started at http://127.0.0.1:41915/ using document root <none> and password file <none>
I20260812 01:03:55.992412 25969 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:55.992566 25969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:55.992901 25969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:55.995973 25969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "0e8ed570b2de475b9b4635fd438a95c5"
format_stamp: "Formatted at 2026-08-12 01:03:55 on dist-test-slave-7f01"
server_key: "efd53c0de1fd3817a93860a58d952e57"
server_key_iv: "4678b1d968edcdb403e4a90c5019702c"
server_key_version: "encryptionkey@0"
I20260812 01:03:55.996762 25969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "0e8ed570b2de475b9b4635fd438a95c5"
format_stamp: "Formatted at 2026-08-12 01:03:55 on dist-test-slave-7f01"
server_key: "efd53c0de1fd3817a93860a58d952e57"
server_key_iv: "4678b1d968edcdb403e4a90c5019702c"
server_key_version: "encryptionkey@0"
I20260812 01:03:56.001986 25969 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:03:56.005566 26001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:56.007004 25969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 01:03:56.007211 25969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "0e8ed570b2de475b9b4635fd438a95c5"
format_stamp: "Formatted at 2026-08-12 01:03:55 on dist-test-slave-7f01"
server_key: "efd53c0de1fd3817a93860a58d952e57"
server_key_iv: "4678b1d968edcdb403e4a90c5019702c"
server_key_version: "encryptionkey@0"
I20260812 01:03:56.007426 25969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:56.013453 25969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:56.017628 25969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:56.017942 25969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:56.045953 25969 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40391
I20260812 01:03:56.045997 26100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40391 every 8 connection(s)
I20260812 01:03:56.047461 25969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:03:56.051438 26102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:03:56.053864 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 25969
I20260812 01:03:56.054082 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:03:56.054563 22309 external_mini_cluster.cc:1428] Setting key c5ff1627cbd7123d83124a8fa7bf047d
I20260812 01:03:56.063864 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:56.056493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52723 (local address 127.0.0.1:40391)
0812 01:03:56.057407 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:56.057427 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:56.057486 (+ 59us) server_negotiation.cc:407] Connection header received
0812 01:03:56.059905 (+ 2419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:56.059934 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:56.060389 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:56.060841 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:56.061856 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:56.061864 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:56.061880 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:56.061897 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:56.061986 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:56.062012 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:56.062247 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:56.062393 (+ 146us) server_negotiation.cc:299] Negotiation successful
0812 01:03:56.062583 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":410,"threads_started":1}
I20260812 01:03:56.065413 26102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e8ed570b2de475b9b4635fd438a95c5: Bootstrap starting.
I20260812 01:03:56.067826 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:03:56.068967 26102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e8ed570b2de475b9b4635fd438a95c5: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:56.070019 26102 log.cc:824] T 00000000000000000000000000000000 P 0e8ed570b2de475b9b4635fd438a95c5: Log is configured to *not* fsync() on all Append() calls
I20260812 01:03:56.072996 26102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e8ed570b2de475b9b4635fd438a95c5: No bootstrap required, opened a new log
I20260812 01:03:56.075171 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:56.072659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56353 (local address 127.0.0.1:40391)
0812 01:03:56.072835 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:56.072841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:56.073199 (+ 358us) server_negotiation.cc:407] Connection header received
0812 01:03:56.073344 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:56.073349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:56.073409 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:56.073536 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:56.074494 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:56.074500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:56.074504 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:03:56.074523 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:56.074568 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:03:56.074587 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:56.074719 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:56.074789 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 01:03:56.074859 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 01:03:56.081341 26102 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e8ed570b2de475b9b4635fd438a95c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8ed570b2de475b9b4635fd438a95c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40391 } }
I20260812 01:03:56.081712 26102 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e8ed570b2de475b9b4635fd438a95c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:56.081847 26102 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e8ed570b2de475b9b4635fd438a95c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e8ed570b2de475b9b4635fd438a95c5, State: Initialized, Role: FOLLOWER
I20260812 01:03:56.082391 26102 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e8ed570b2de475b9b4635fd438a95c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8ed570b2de475b9b4635fd438a95c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40391 } }
I20260812 01:03:56.082647 26102 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e8ed570b2de475b9b4635fd438a95c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:03:56.082791 26102 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e8ed570b2de475b9b4635fd438a95c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:03:56.082964 26102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e8ed570b2de475b9b4635fd438a95c5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:56.085253 26102 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e8ed570b2de475b9b4635fd438a95c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8ed570b2de475b9b4635fd438a95c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40391 } }
I20260812 01:03:56.085760 26102 leader_election.cc:302] T 00000000000000000000000000000000 P 0e8ed570b2de475b9b4635fd438a95c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e8ed570b2de475b9b4635fd438a95c5; no voters:
I20260812 01:03:56.086375 26102 leader_election.cc:288] T 00000000000000000000000000000000 P 0e8ed570b2de475b9b4635fd438a95c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:03:56.086532 26118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e8ed570b2de475b9b4635fd438a95c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:03:56.087659 26118 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e8ed570b2de475b9b4635fd438a95c5 [term 1 LEADER]: Becoming Leader. State: Replica: 0e8ed570b2de475b9b4635fd438a95c5, State: Running, Role: LEADER
I20260812 01:03:56.088272 26118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e8ed570b2de475b9b4635fd438a95c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8ed570b2de475b9b4635fd438a95c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40391 } }
I20260812 01:03:56.088668 26102 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e8ed570b2de475b9b4635fd438a95c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:03:56.091323 26120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e8ed570b2de475b9b4635fd438a95c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e8ed570b2de475b9b4635fd438a95c5. Latest consensus state: current_term: 1 leader_uuid: "0e8ed570b2de475b9b4635fd438a95c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8ed570b2de475b9b4635fd438a95c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40391 } } }
I20260812 01:03:56.091418 26119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e8ed570b2de475b9b4635fd438a95c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e8ed570b2de475b9b4635fd438a95c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8ed570b2de475b9b4635fd438a95c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40391 } } }
I20260812 01:03:56.091810 26120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e8ed570b2de475b9b4635fd438a95c5 [sys.catalog]: This master's current role is: LEADER
I20260812 01:03:56.091809 26119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e8ed570b2de475b9b4635fd438a95c5 [sys.catalog]: This master's current role is: LEADER
I20260812 01:03:56.094050 26128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:03:56.100589 26128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:03:56.110185 26128 catalog_manager.cc:1347] Generated new cluster ID: 83d47af39dfe4f7f8a235bca0922d156
I20260812 01:03:56.110306 26128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:03:56.144644 26128 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:03:56.145090 26128 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:03:56.157545 26128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e8ed570b2de475b9b4635fd438a95c5: Generated new TSK 0
I20260812 01:03:56.158291 26128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:03:56.170364 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 25969
W20260812 01:03:56.171433 26094 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52723
W20260812 01:03:56.172161 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 25969: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:03:56.172310 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 25969
2026-08-12T01:03:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2296 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:03:56 dist-test-slave-7f01 krb5kdc[26158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:03:56 dist-test-slave-7f01 krb5kdc[26158](info): set up 2 sockets
Aug 12 01:03:56 dist-test-slave-7f01 krb5kdc[26158](info): commencing operation
krb5kdc: starting...
W20260812 01:03:57.560390 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.342s user 0.000s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:03:57 dist-test-slave-7f01 krb5kdc[26158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:03:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:03:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 01:03:57.875036 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38331
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:03:58.228289 26177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:03:58.228600 26177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:03:58.228777 26177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:03:58.236130 26177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:03:58.236251 26177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:03:58.236351 26177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:03:58.236470 26177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:03:58.236554 26177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:03:58.246824 26177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:03:58.248250 26177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:03:58.249684 26177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:03:58.262319 26187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:58.262409 26188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:03:58.262362 26190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:03:58.263722 26177 server_base.cc:1034] running on GCE node
I20260812 01:03:58.264642 26177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:03:58.266135 26177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:03:58.267376 26177 hybrid_clock.cc:648] HybridClock initialized: now 1786496638267340 us; error 50 us; skew 500 ppm
I20260812 01:03:58.267799 26177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:03:58 dist-test-slave-7f01 krb5kdc[26158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:03:58.276065 26177 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 01:03:58.277778 26177 webserver.cc:460] Webserver started at http://127.0.0.1:41659/ using document root <none> and password file <none>
I20260812 01:03:58.278503 26177 fs_manager.cc:359] Metadata directory not provided
I20260812 01:03:58.278661 26177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:03:58.278995 26177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:03:58.282204 26177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "06c7259dc02147e781faedadd0710664"
format_stamp: "Formatted at 2026-08-12 01:03:58 on dist-test-slave-7f01"
server_key: "ba5a8cfff16c4c21f8d719cacb3573ce"
server_key_iv: "ce645da8a63dbb38d9631f8279c51b09"
server_key_version: "encryptionkey@0"
I20260812 01:03:58.283059 26177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "06c7259dc02147e781faedadd0710664"
format_stamp: "Formatted at 2026-08-12 01:03:58 on dist-test-slave-7f01"
server_key: "ba5a8cfff16c4c21f8d719cacb3573ce"
server_key_iv: "ce645da8a63dbb38d9631f8279c51b09"
server_key_version: "encryptionkey@0"
I20260812 01:03:58.288470 26177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 01:03:58.292061 26204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:03:58.293366 26177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 01:03:58.293574 26177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "06c7259dc02147e781faedadd0710664"
format_stamp: "Formatted at 2026-08-12 01:03:58 on dist-test-slave-7f01"
server_key: "ba5a8cfff16c4c21f8d719cacb3573ce"
server_key_iv: "ce645da8a63dbb38d9631f8279c51b09"
server_key_version: "encryptionkey@0"
I20260812 01:03:58.293783 26177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:03:58.314667 26177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:03:58.319902 26177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:03:58.320223 26177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:03:58.340379 26177 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38331
I20260812 01:03:58.340412 26293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38331 every 8 connection(s)
I20260812 01:03:58.341791 26177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:03:58.345642 26295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:03:58.348740 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 26177
I20260812 01:03:58.348953 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:03:58.349535 22309 external_mini_cluster.cc:1428] Setting key 9070a6d5db46660bd2fd33e0e11f59e4
I20260812 01:03:58.358701 26295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06c7259dc02147e781faedadd0710664: Bootstrap starting.
I20260812 01:03:58.362965 26295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06c7259dc02147e781faedadd0710664: Neither blocks nor log segments found. Creating new log.
I20260812 01:03:58.364459 26295 log.cc:824] T 00000000000000000000000000000000 P 06c7259dc02147e781faedadd0710664: Log is configured to *not* fsync() on all Append() calls
Aug 12 01:03:58 dist-test-slave-7f01 krb5kdc[26158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 01:03:58.367830 26295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06c7259dc02147e781faedadd0710664: No bootstrap required, opened a new log
I20260812 01:03:58.374873 26302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:58.351493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46679 (local address 127.0.0.1:38331)
0812 01:03:58.352405 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:58.352427 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:58.352470 (+ 43us) server_negotiation.cc:407] Connection header received
0812 01:03:58.354243 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:58.354266 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:58.354665 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:58.355003 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:58.357145 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:58.358491 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:58.360427 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:58.361203 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:58.361351 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:58.366898 (+ 5547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:58.366906 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:58.366921 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:58.366965 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:58.370382 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:58.371196 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:58.371203 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:58.371209 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:58.371395 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:58.371871 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:58.371875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:58.371878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:58.372488 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:58.372843 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:58.373370 (+ 527us) server_negotiation.cc:299] Negotiation successful
0812 01:03:58.373807 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":474,"threads_started":1}
I20260812 01:03:58.376435 26295 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06c7259dc02147e781faedadd0710664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c7259dc02147e781faedadd0710664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38331 } }
I20260812 01:03:58.376838 26295 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06c7259dc02147e781faedadd0710664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:03:58.376972 26295 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06c7259dc02147e781faedadd0710664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c7259dc02147e781faedadd0710664, State: Initialized, Role: FOLLOWER
I20260812 01:03:58.377580 26295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06c7259dc02147e781faedadd0710664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c7259dc02147e781faedadd0710664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38331 } }
I20260812 01:03:58.377846 26295 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06c7259dc02147e781faedadd0710664 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:03:58.378018 26295 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06c7259dc02147e781faedadd0710664 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:03:58.378228 26295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06c7259dc02147e781faedadd0710664 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:03:58.378949 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:03:58.380787 26295 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06c7259dc02147e781faedadd0710664 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c7259dc02147e781faedadd0710664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38331 } }
I20260812 01:03:58.381314 26295 leader_election.cc:302] T 00000000000000000000000000000000 P 06c7259dc02147e781faedadd0710664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06c7259dc02147e781faedadd0710664; no voters:
I20260812 01:03:58.382107 26295 leader_election.cc:288] T 00000000000000000000000000000000 P 06c7259dc02147e781faedadd0710664 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:03:58.382239 26309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06c7259dc02147e781faedadd0710664 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:03:58.385551 26309 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06c7259dc02147e781faedadd0710664 [term 1 LEADER]: Becoming Leader. State: Replica: 06c7259dc02147e781faedadd0710664, State: Running, Role: LEADER
I20260812 01:03:58.385967 26295 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06c7259dc02147e781faedadd0710664 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:03:58.386243 26309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06c7259dc02147e781faedadd0710664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c7259dc02147e781faedadd0710664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38331 } }
I20260812 01:03:58.389708 26315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06c7259dc02147e781faedadd0710664 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06c7259dc02147e781faedadd0710664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c7259dc02147e781faedadd0710664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38331 } } }
I20260812 01:03:58.390579 26316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06c7259dc02147e781faedadd0710664 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06c7259dc02147e781faedadd0710664. Latest consensus state: current_term: 1 leader_uuid: "06c7259dc02147e781faedadd0710664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c7259dc02147e781faedadd0710664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38331 } } }
I20260812 01:03:58.391366 26316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06c7259dc02147e781faedadd0710664 [sys.catalog]: This master's current role is: LEADER
I20260812 01:03:58.396229 26315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06c7259dc02147e781faedadd0710664 [sys.catalog]: This master's current role is: LEADER
I20260812 01:03:58.399042 26330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:03:58.410996 26302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:03:58.384840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55235 (local address 127.0.0.1:38331)
0812 01:03:58.385031 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 01:03:58.385036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:03:58.385051 (+ 15us) server_negotiation.cc:407] Connection header received
0812 01:03:58.385095 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:03:58.385099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:03:58.385158 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:03:58.385270 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:03:58.387161 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:58.388500 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:58.393675 (+ 5175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:03:58.394545 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:03:58.395320 (+ 775us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:03:58.398884 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:03:58.398893 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:03:58.398899 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:03:58.398945 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:03:58.406265 (+ 7320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:58.407474 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:58.407480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:58.407483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:58.407656 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:03:58.408213 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:03:58.408218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:03:58.408221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:03:58.408699 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:03:58.408964 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0812 01:03:58.410368 (+ 1404us) server_negotiation.cc:299] Negotiation successful
0812 01:03:58.410612 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 01:03:58.411631 26330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:03:58.422809 26330 catalog_manager.cc:1347] Generated new cluster ID: 2947e5d6b29c4f5eb54bd1033202fa5b
I20260812 01:03:58.422941 26330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:03:58.498749 26330 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:03:58.500221 26330 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:03:58.516126 26330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06c7259dc02147e781faedadd0710664: Generated new TSK 0
I20260812 01:03:58.516996 26330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:03:58.530287 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 26177
I20260812 01:03:58.531400 26286 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:03:58.635310 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 26177
2026-08-12T01:03:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2461 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:03:58 dist-test-slave-7f01 krb5kdc[26347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:03:58 dist-test-slave-7f01 krb5kdc[26347](info): set up 2 sockets
Aug 12 01:03:58 dist-test-slave-7f01 krb5kdc[26347](info): commencing operation
krb5kdc: starting...
W20260812 01:04:00.026026 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.346s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:04:00 dist-test-slave-7f01 krb5kdc[26347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:04:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:04:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 01:04:00.344002 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42237
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:04:00.693420 26372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:04:00.693725 26372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:04:00.693835 26372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:04:00.701112 26372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:04:00.701293 26372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:04:00.701406 26372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:04:00.701519 26372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:04:00.701599 26372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:04:00.711474 26372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:04:00.712855 26372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:04:00.714191 26372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:04:00.726739 26377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:00.726972 26379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:00.727322 26382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:04:00.728098 26372 server_base.cc:1034] running on GCE node
I20260812 01:04:00.729008 26372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:04:00.730415 26372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:04:00.731637 26372 hybrid_clock.cc:648] HybridClock initialized: now 1786496640731587 us; error 34 us; skew 500 ppm
I20260812 01:04:00.732116 26372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:04:00 dist-test-slave-7f01 krb5kdc[26347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:04:00.735612 26372 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 01:04:00.737217 26372 webserver.cc:460] Webserver started at http://127.0.0.1:41815/ using document root <none> and password file <none>
I20260812 01:04:00.737901 26372 fs_manager.cc:359] Metadata directory not provided
I20260812 01:04:00.738040 26372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:04:00.738368 26372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:04:00.741499 26372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "b0ad5519364a450fa8784f60d2209679"
format_stamp: "Formatted at 2026-08-12 01:04:00 on dist-test-slave-7f01"
server_key: "60f460d75203283e5dc408428170ebad"
server_key_iv: "7bcb7dafd2627edaff9ef905b10a1506"
server_key_version: "encryptionkey@0"
I20260812 01:04:00.742311 26372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "b0ad5519364a450fa8784f60d2209679"
format_stamp: "Formatted at 2026-08-12 01:04:00 on dist-test-slave-7f01"
server_key: "60f460d75203283e5dc408428170ebad"
server_key_iv: "7bcb7dafd2627edaff9ef905b10a1506"
server_key_version: "encryptionkey@0"
I20260812 01:04:00.747495 26372 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 01:04:00.751334 26395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:04:00.752683 26372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:04:00.752894 26372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "b0ad5519364a450fa8784f60d2209679"
format_stamp: "Formatted at 2026-08-12 01:04:00 on dist-test-slave-7f01"
server_key: "60f460d75203283e5dc408428170ebad"
server_key_iv: "7bcb7dafd2627edaff9ef905b10a1506"
server_key_version: "encryptionkey@0"
I20260812 01:04:00.753109 26372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:04:00.765530 26372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:04:00.769682 26372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:04:00.769979 26372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:04:00.789709 26372 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42237
I20260812 01:04:00.789770 26485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42237 every 8 connection(s)
I20260812 01:04:00.791182 26372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:04:00.794878 26486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:04:00.795825 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 26372
I20260812 01:04:00.796022 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:04:00.796636 22309 external_mini_cluster.cc:1428] Setting key 4ade4afd7829021477ee2268ab5ac187
I20260812 01:04:00.809624 26486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ad5519364a450fa8784f60d2209679: Bootstrap starting.
I20260812 01:04:00.811295 26490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:00.798596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56241 (local address 127.0.0.1:42237)
0812 01:04:00.799410 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:00.799423 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:00.799464 (+ 41us) server_negotiation.cc:407] Connection header received
0812 01:04:00.801755 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:00.801784 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:00.802209 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:00.802593 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:00.803218 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:00.804744 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:00.806757 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:00.807513 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:00.807653 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:00.809004 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:00.809011 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:00.809025 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:04:00.809567 (+ 542us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:00.809647 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:04:00.809673 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:00.809832 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:00.810041 (+ 209us) server_negotiation.cc:299] Negotiation successful
0812 01:04:00.810313 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":387,"threads_started":1}
I20260812 01:04:00.813302 26486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0ad5519364a450fa8784f60d2209679: Neither blocks nor log segments found. Creating new log.
I20260812 01:04:00.814558 26486 log.cc:824] T 00000000000000000000000000000000 P b0ad5519364a450fa8784f60d2209679: Log is configured to *not* fsync() on all Append() calls
I20260812 01:04:00.815465 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:04:00.817667 26486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ad5519364a450fa8784f60d2209679: No bootstrap required, opened a new log
I20260812 01:04:00.826880 26486 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0ad5519364a450fa8784f60d2209679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ad5519364a450fa8784f60d2209679" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42237 } }
I20260812 01:04:00.827441 26486 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0ad5519364a450fa8784f60d2209679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:04:00.827606 26486 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0ad5519364a450fa8784f60d2209679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ad5519364a450fa8784f60d2209679, State: Initialized, Role: FOLLOWER
I20260812 01:04:00.827874 26490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:00.820816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34677 (local address 127.0.0.1:42237)
0812 01:04:00.821012 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:00.821018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:00.821502 (+ 484us) server_negotiation.cc:407] Connection header received
0812 01:04:00.821661 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:00.821667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:00.821758 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:00.821873 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:00.822369 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:00.823537 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:00.824956 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:00.825676 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:00.825807 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:00.826990 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:00.826997 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:00.827002 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:04:00.827025 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:00.827077 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:04:00.827098 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:00.827205 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:00.827348 (+ 143us) server_negotiation.cc:299] Negotiation successful
0812 01:04:00.827471 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 01:04:00.828379 26486 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0ad5519364a450fa8784f60d2209679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ad5519364a450fa8784f60d2209679" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42237 } }
I20260812 01:04:00.828661 26486 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0ad5519364a450fa8784f60d2209679 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:04:00.828851 26486 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0ad5519364a450fa8784f60d2209679 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:04:00.829035 26486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0ad5519364a450fa8784f60d2209679 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:04:00.831353 26486 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0ad5519364a450fa8784f60d2209679 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ad5519364a450fa8784f60d2209679" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42237 } }
I20260812 01:04:00.831946 26486 leader_election.cc:302] T 00000000000000000000000000000000 P b0ad5519364a450fa8784f60d2209679 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0ad5519364a450fa8784f60d2209679; no voters:
I20260812 01:04:00.832597 26486 leader_election.cc:288] T 00000000000000000000000000000000 P b0ad5519364a450fa8784f60d2209679 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:04:00.832742 26500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0ad5519364a450fa8784f60d2209679 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:04:00.833572 26500 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0ad5519364a450fa8784f60d2209679 [term 1 LEADER]: Becoming Leader. State: Replica: b0ad5519364a450fa8784f60d2209679, State: Running, Role: LEADER
I20260812 01:04:00.834266 26500 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0ad5519364a450fa8784f60d2209679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ad5519364a450fa8784f60d2209679" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42237 } }
I20260812 01:04:00.835057 26486 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0ad5519364a450fa8784f60d2209679 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:04:00.837450 26507 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ad5519364a450fa8784f60d2209679 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0ad5519364a450fa8784f60d2209679. Latest consensus state: current_term: 1 leader_uuid: "b0ad5519364a450fa8784f60d2209679" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ad5519364a450fa8784f60d2209679" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42237 } } }
I20260812 01:04:00.837857 26507 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ad5519364a450fa8784f60d2209679 [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:00.837563 26503 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ad5519364a450fa8784f60d2209679 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0ad5519364a450fa8784f60d2209679" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ad5519364a450fa8784f60d2209679" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42237 } } }
I20260812 01:04:00.837996 26503 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ad5519364a450fa8784f60d2209679 [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:00.840147 26511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:04:00.846788 26511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:04:00.857041 26511 catalog_manager.cc:1347] Generated new cluster ID: ea9a1cd4cb1f4c858cf34866887c74a8
I20260812 01:04:00.857201 26511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:04:00.878294 26511 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:04:00.880450 26511 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:04:00.896919 26511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0ad5519364a450fa8784f60d2209679: Generated new TSK 0
I20260812 01:04:00.898011 26511 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:04:00.908044 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 26372
W20260812 01:04:00.909382 26476 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56241
W20260812 01:04:00.910115 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 26372: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:04:00.910274 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 26372
2026-08-12T01:04:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2272 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:04:01 dist-test-slave-7f01 krb5kdc[26533](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:04:01 dist-test-slave-7f01 krb5kdc[26533](info): set up 2 sockets
Aug 12 01:04:01 dist-test-slave-7f01 krb5kdc[26533](info): commencing operation
krb5kdc: starting...
W20260812 01:04:02.288014 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.336s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:04:02 dist-test-slave-7f01 krb5kdc[26533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:04:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:04:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 01:04:02.617684 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:04:02.978441 26566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:04:02.978830 26566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:04:02.979012 26566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:04:02.986322 26566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:04:02.986492 26566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:04:02.986610 26566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:04:02.986727 26566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:04:02.986831 26566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:04:02.996978 26566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:04:02.998346 26566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:04:02.999704 26566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:04:03.011926 26574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:03.011879 26577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:03.012022 26579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:04:03.013170 26566 server_base.cc:1034] running on GCE node
I20260812 01:04:03.014061 26566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:04:03.015520 26566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:04:03.016780 26566 hybrid_clock.cc:648] HybridClock initialized: now 1786496643016730 us; error 45 us; skew 500 ppm
I20260812 01:04:03.017163 26566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:04:03 dist-test-slave-7f01 krb5kdc[26533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:04:03.020582 26566 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 01:04:03.022193 26566 webserver.cc:460] Webserver started at http://127.0.0.1:43285/ using document root <none> and password file <none>
I20260812 01:04:03.022902 26566 fs_manager.cc:359] Metadata directory not provided
I20260812 01:04:03.023064 26566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:04:03.023393 26566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:04:03.026585 26566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "94b319ae3c344701b1174ae8a8a1ebb8"
format_stamp: "Formatted at 2026-08-12 01:04:03 on dist-test-slave-7f01"
server_key: "45b733f8a23514cc94c011605f241943"
server_key_iv: "26746f0912c9b402a69c904b540d9b12"
server_key_version: "encryptionkey@0"
I20260812 01:04:03.027418 26566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "94b319ae3c344701b1174ae8a8a1ebb8"
format_stamp: "Formatted at 2026-08-12 01:04:03 on dist-test-slave-7f01"
server_key: "45b733f8a23514cc94c011605f241943"
server_key_iv: "26746f0912c9b402a69c904b540d9b12"
server_key_version: "encryptionkey@0"
I20260812 01:04:03.032879 26566 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 01:04:03.036547 26594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:04:03.037928 26566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:04:03.038133 26566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "94b319ae3c344701b1174ae8a8a1ebb8"
format_stamp: "Formatted at 2026-08-12 01:04:03 on dist-test-slave-7f01"
server_key: "45b733f8a23514cc94c011605f241943"
server_key_iv: "26746f0912c9b402a69c904b540d9b12"
server_key_version: "encryptionkey@0"
I20260812 01:04:03.038350 26566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:04:03.044401 26566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:04:03.048802 26566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:04:03.049142 26566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:04:03.069092 26566 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45305
I20260812 01:04:03.069142 26678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45305 every 8 connection(s)
I20260812 01:04:03.070588 26566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:04:03.071624 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 26566
I20260812 01:04:03.071907 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:04:03.072517 22309 external_mini_cluster.cc:1428] Setting key 6f9d19d2881f3ee6beea3b4a750e3369
I20260812 01:04:03.075464 26679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:04:03.081244 26694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:03.074495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38645 (local address 127.0.0.1:45305)
0812 01:04:03.075325 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:03.075338 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:03.075380 (+ 42us) server_negotiation.cc:407] Connection header received
0812 01:04:03.077267 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:03.077291 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:03.077631 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:03.078161 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:03.079435 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:03.079441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:03.079454 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:04:03.079487 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:03.079563 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:04:03.079583 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:03.079869 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:03.080028 (+ 159us) server_negotiation.cc:299] Negotiation successful
0812 01:04:03.080220 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":378,"threads_started":1}
I20260812 01:04:03.085058 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:04:03.087471 26679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94b319ae3c344701b1174ae8a8a1ebb8: Bootstrap starting.
I20260812 01:04:03.091262 26679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94b319ae3c344701b1174ae8a8a1ebb8: Neither blocks nor log segments found. Creating new log.
I20260812 01:04:03.092434 26694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:03.089792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45123 (local address 127.0.0.1:45305)
0812 01:04:03.090035 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:03.090041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:03.090330 (+ 289us) server_negotiation.cc:407] Connection header received
0812 01:04:03.090515 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:03.090518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:03.090570 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:03.090650 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:03.091686 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:03.091690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:03.091697 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:04:03.091711 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:03.091809 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:04:03.091839 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:03.091933 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:03.092080 (+ 147us) server_negotiation.cc:299] Negotiation successful
0812 01:04:03.092130 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260812 01:04:03.092770 26679 log.cc:824] T 00000000000000000000000000000000 P 94b319ae3c344701b1174ae8a8a1ebb8: Log is configured to *not* fsync() on all Append() calls
I20260812 01:04:03.095613 26679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94b319ae3c344701b1174ae8a8a1ebb8: No bootstrap required, opened a new log
I20260812 01:04:03.103040 26679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94b319ae3c344701b1174ae8a8a1ebb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b319ae3c344701b1174ae8a8a1ebb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45305 } }
I20260812 01:04:03.103464 26679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94b319ae3c344701b1174ae8a8a1ebb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:04:03.103595 26679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94b319ae3c344701b1174ae8a8a1ebb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94b319ae3c344701b1174ae8a8a1ebb8, State: Initialized, Role: FOLLOWER
I20260812 01:04:03.104300 26679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94b319ae3c344701b1174ae8a8a1ebb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b319ae3c344701b1174ae8a8a1ebb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45305 } }
I20260812 01:04:03.104545 26679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94b319ae3c344701b1174ae8a8a1ebb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:04:03.104722 26679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94b319ae3c344701b1174ae8a8a1ebb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:04:03.104902 26679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94b319ae3c344701b1174ae8a8a1ebb8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:04:03.107260 26679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94b319ae3c344701b1174ae8a8a1ebb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b319ae3c344701b1174ae8a8a1ebb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45305 } }
I20260812 01:04:03.107883 26679 leader_election.cc:302] T 00000000000000000000000000000000 P 94b319ae3c344701b1174ae8a8a1ebb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94b319ae3c344701b1174ae8a8a1ebb8; no voters:
I20260812 01:04:03.108495 26679 leader_election.cc:288] T 00000000000000000000000000000000 P 94b319ae3c344701b1174ae8a8a1ebb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:04:03.108657 26705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94b319ae3c344701b1174ae8a8a1ebb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:04:03.109440 26705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94b319ae3c344701b1174ae8a8a1ebb8 [term 1 LEADER]: Becoming Leader. State: Replica: 94b319ae3c344701b1174ae8a8a1ebb8, State: Running, Role: LEADER
I20260812 01:04:03.109966 26705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94b319ae3c344701b1174ae8a8a1ebb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b319ae3c344701b1174ae8a8a1ebb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45305 } }
I20260812 01:04:03.110769 26679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94b319ae3c344701b1174ae8a8a1ebb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:04:03.113011 26707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94b319ae3c344701b1174ae8a8a1ebb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94b319ae3c344701b1174ae8a8a1ebb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b319ae3c344701b1174ae8a8a1ebb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45305 } } }
I20260812 01:04:03.113370 26707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94b319ae3c344701b1174ae8a8a1ebb8 [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:03.113763 26709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94b319ae3c344701b1174ae8a8a1ebb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94b319ae3c344701b1174ae8a8a1ebb8. Latest consensus state: current_term: 1 leader_uuid: "94b319ae3c344701b1174ae8a8a1ebb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b319ae3c344701b1174ae8a8a1ebb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45305 } } }
I20260812 01:04:03.114158 26709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94b319ae3c344701b1174ae8a8a1ebb8 [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:03.116642 26720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:04:03.121973 26720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:04:03.130561 26720 catalog_manager.cc:1347] Generated new cluster ID: 786d7699b67440d2960a6e17a5e6444b
I20260812 01:04:03.130702 26720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:04:03.143674 26720 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:04:03.144150 26720 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:04:03.153693 26720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94b319ae3c344701b1174ae8a8a1ebb8: Generated new TSK 0
I20260812 01:04:03.154475 26720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:04:03.171283 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 26566
W20260812 01:04:03.172403 26673 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38645
W20260812 01:04:03.173127 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 26566: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:04:03.173269 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 26566
2026-08-12T01:04:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2268 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:04:03 dist-test-slave-7f01 krb5kdc[26744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:04:03 dist-test-slave-7f01 krb5kdc[26744](info): set up 2 sockets
Aug 12 01:04:03 dist-test-slave-7f01 krb5kdc[26744](info): commencing operation
krb5kdc: starting...
W20260812 01:04:04.571313 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.350s user 0.000s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:04:04 dist-test-slave-7f01 krb5kdc[26744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:04:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:04:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 01:04:04.907601 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33481
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:04:05.261607 26764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:04:05.261988 26764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:04:05.262162 26764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:04:05.269809 26764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:04:05.269990 26764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:04:05.270098 26764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:04:05.270212 26764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:04:05.270296 26764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:04:05.280668 26764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:04:05.282104 26764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:04:05.283447 26764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:04:05.296000 26772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:05.296082 26775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:05.296105 26771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:04:05.297060 26764 server_base.cc:1034] running on GCE node
I20260812 01:04:05.297928 26764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:04:05.299516 26764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:04:05.300753 26764 hybrid_clock.cc:648] HybridClock initialized: now 1786496645300692 us; error 56 us; skew 500 ppm
I20260812 01:04:05.301091 26764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:04:05 dist-test-slave-7f01 krb5kdc[26744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:04:05.312951 26764 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 01:04:05.314622 26764 webserver.cc:460] Webserver started at http://127.0.0.1:41271/ using document root <none> and password file <none>
I20260812 01:04:05.315318 26764 fs_manager.cc:359] Metadata directory not provided
I20260812 01:04:05.315431 26764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:04:05.315711 26764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:04:05.318949 26764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "9de70ca5c6a145058a39b7f33cd3bbef"
format_stamp: "Formatted at 2026-08-12 01:04:05 on dist-test-slave-7f01"
server_key: "471976ea9396cd83d748be1767bd3950"
server_key_iv: "c354c46fb516b22006c7114d3fb49aa9"
server_key_version: "encryptionkey@0"
I20260812 01:04:05.319720 26764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "9de70ca5c6a145058a39b7f33cd3bbef"
format_stamp: "Formatted at 2026-08-12 01:04:05 on dist-test-slave-7f01"
server_key: "471976ea9396cd83d748be1767bd3950"
server_key_iv: "c354c46fb516b22006c7114d3fb49aa9"
server_key_version: "encryptionkey@0"
I20260812 01:04:05.325007 26764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260812 01:04:05.328519 26785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:04:05.329841 26764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:04:05.330034 26764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "9de70ca5c6a145058a39b7f33cd3bbef"
format_stamp: "Formatted at 2026-08-12 01:04:05 on dist-test-slave-7f01"
server_key: "471976ea9396cd83d748be1767bd3950"
server_key_iv: "c354c46fb516b22006c7114d3fb49aa9"
server_key_version: "encryptionkey@0"
I20260812 01:04:05.330245 26764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:04:05.347968 26764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:04:05.352293 26764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:04:05.352605 26764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:04:05.372511 26764 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33481
I20260812 01:04:05.372550 26866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33481 every 8 connection(s)
I20260812 01:04:05.373939 26764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:04:05.377754 26867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:04:05.380872 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 26764
I20260812 01:04:05.381074 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:04:05.381587 22309 external_mini_cluster.cc:1428] Setting key 6d335cc0b9bce7a9fd62943d4d97137a
I20260812 01:04:05.390362 26867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9de70ca5c6a145058a39b7f33cd3bbef: Bootstrap starting.
I20260812 01:04:05.394367 26867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9de70ca5c6a145058a39b7f33cd3bbef: Neither blocks nor log segments found. Creating new log.
I20260812 01:04:05.395799 26867 log.cc:824] T 00000000000000000000000000000000 P 9de70ca5c6a145058a39b7f33cd3bbef: Log is configured to *not* fsync() on all Append() calls
Aug 12 01:04:05 dist-test-slave-7f01 krb5kdc[26744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 01:04:05.399302 26867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9de70ca5c6a145058a39b7f33cd3bbef: No bootstrap required, opened a new log
I20260812 01:04:05.407353 26872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:05.383421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:42063 (local address 127.0.0.1:33481)
0812 01:04:05.384223 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:05.384234 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:05.384278 (+ 44us) server_negotiation.cc:407] Connection header received
0812 01:04:05.386040 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:05.386063 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:05.386461 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:05.386795 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:05.388856 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:05.390194 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:05.392048 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:05.392728 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:05.392854 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:05.398633 (+ 5779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:05.398642 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:05.398657 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:05.398696 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:05.402478 (+ 3782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:05.403532 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:05.403539 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:05.403546 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:05.403730 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:05.404329 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:05.404333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:05.404335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:05.404956 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:05.405221 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:05.405745 (+ 524us) server_negotiation.cc:299] Negotiation successful
0812 01:04:05.406164 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":389,"threads_started":1}
I20260812 01:04:05.408785 26867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9de70ca5c6a145058a39b7f33cd3bbef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de70ca5c6a145058a39b7f33cd3bbef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } }
I20260812 01:04:05.409226 26867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9de70ca5c6a145058a39b7f33cd3bbef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:04:05.409363 26867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9de70ca5c6a145058a39b7f33cd3bbef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9de70ca5c6a145058a39b7f33cd3bbef, State: Initialized, Role: FOLLOWER
I20260812 01:04:05.410039 26867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9de70ca5c6a145058a39b7f33cd3bbef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de70ca5c6a145058a39b7f33cd3bbef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } }
I20260812 01:04:05.410398 26867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9de70ca5c6a145058a39b7f33cd3bbef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:04:05.410557 26867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9de70ca5c6a145058a39b7f33cd3bbef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:04:05.410751 26867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9de70ca5c6a145058a39b7f33cd3bbef [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:04:05.411969 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:04:05.413666 26867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9de70ca5c6a145058a39b7f33cd3bbef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de70ca5c6a145058a39b7f33cd3bbef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } }
I20260812 01:04:05.414288 26867 leader_election.cc:302] T 00000000000000000000000000000000 P 9de70ca5c6a145058a39b7f33cd3bbef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9de70ca5c6a145058a39b7f33cd3bbef; no voters:
I20260812 01:04:05.415069 26867 leader_election.cc:288] T 00000000000000000000000000000000 P 9de70ca5c6a145058a39b7f33cd3bbef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:04:05.415212 26875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9de70ca5c6a145058a39b7f33cd3bbef [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:04:05.416949 26875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9de70ca5c6a145058a39b7f33cd3bbef [term 1 LEADER]: Becoming Leader. State: Replica: 9de70ca5c6a145058a39b7f33cd3bbef, State: Running, Role: LEADER
I20260812 01:04:05.417691 26875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9de70ca5c6a145058a39b7f33cd3bbef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de70ca5c6a145058a39b7f33cd3bbef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } }
I20260812 01:04:05.417915 26867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9de70ca5c6a145058a39b7f33cd3bbef [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:04:05.421648 26879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9de70ca5c6a145058a39b7f33cd3bbef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9de70ca5c6a145058a39b7f33cd3bbef. Latest consensus state: current_term: 1 leader_uuid: "9de70ca5c6a145058a39b7f33cd3bbef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de70ca5c6a145058a39b7f33cd3bbef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } } }
I20260812 01:04:05.422238 26879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9de70ca5c6a145058a39b7f33cd3bbef [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:05.423012 26878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9de70ca5c6a145058a39b7f33cd3bbef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9de70ca5c6a145058a39b7f33cd3bbef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de70ca5c6a145058a39b7f33cd3bbef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } } }
I20260812 01:04:05.423391 26878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9de70ca5c6a145058a39b7f33cd3bbef [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:05.434966 26887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:04:05.441603 26887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:04:05.459829 26872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:05.434771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:51955 (local address 127.0.0.1:33481)
0812 01:04:05.435000 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:05.435011 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:05.436365 (+ 1354us) server_negotiation.cc:407] Connection header received
0812 01:04:05.436759 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:05.436769 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:05.436929 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:05.437047 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:05.439562 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:05.441099 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:05.442609 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:05.443568 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:05.443784 (+ 216us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:05.448834 (+ 5050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:05.448847 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:05.448856 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:05.448914 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:05.454300 (+ 5386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:05.455727 (+ 1427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:05.455762 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:05.455769 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:05.455983 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:05.456850 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:05.456860 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:05.456866 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:05.457482 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:05.457842 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:05.458848 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0812 01:04:05.459154 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 01:04:05.463275 26887 catalog_manager.cc:1347] Generated new cluster ID: bce24781ff98420cb6a58c31dcf3eb4b
I20260812 01:04:05.463404 26887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:04:05.523959 26887 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:04:05.525384 26887 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:04:05.537734 26887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9de70ca5c6a145058a39b7f33cd3bbef: Generated new TSK 0
I20260812 01:04:05.538769 26887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:04:05.557605 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 26764
I20260812 01:04:05.559855 26861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:04:05.661140 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 26764
W20260812 01:04:05.688014 26741 connection.cc:535] client connection to 127.0.0.1:33481 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33481): Connection reset by peer (error 104)
2026-08-12T01:04:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2486 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:04:05 dist-test-slave-7f01 krb5kdc[26919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:04:05 dist-test-slave-7f01 krb5kdc[26919](info): set up 2 sockets
Aug 12 01:04:05 dist-test-slave-7f01 krb5kdc[26919](info): commencing operation
krb5kdc: starting...
W20260812 01:04:07.058571 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.351s user 0.008s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:04:07 dist-test-slave-7f01 krb5kdc[26919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:04:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:04:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 01:04:07.388494 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35591
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:04:07.739467 26942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:04:07.739837 26942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:04:07.740046 26942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:04:07.747723 26942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:04:07.747867 26942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:04:07.747963 26942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:04:07.748076 26942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:04:07.748142 26942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:04:07.758070 26942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:04:07.759438 26942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:04:07.760885 26942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:04:07.773267 26951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:07.773447 26955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:07.773313 26952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:04:07.774760 26942 server_base.cc:1034] running on GCE node
I20260812 01:04:07.775624 26942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:04:07.777102 26942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:04:07.778308 26942 hybrid_clock.cc:648] HybridClock initialized: now 1786496647778260 us; error 44 us; skew 500 ppm
I20260812 01:04:07.778699 26942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:04:07 dist-test-slave-7f01 krb5kdc[26919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:04:07.782069 26942 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 01:04:07.783661 26942 webserver.cc:460] Webserver started at http://127.0.0.1:42609/ using document root <none> and password file <none>
I20260812 01:04:07.784412 26942 fs_manager.cc:359] Metadata directory not provided
I20260812 01:04:07.784575 26942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:04:07.784909 26942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:04:07.788069 26942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "4524095b57924b93adb4162b2379f84d"
format_stamp: "Formatted at 2026-08-12 01:04:07 on dist-test-slave-7f01"
server_key: "ef848f4f61c757a4cd27f64a5fc06c1a"
server_key_iv: "52d7a2a65b38f14871f7d284269023f8"
server_key_version: "encryptionkey@0"
I20260812 01:04:07.788893 26942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "4524095b57924b93adb4162b2379f84d"
format_stamp: "Formatted at 2026-08-12 01:04:07 on dist-test-slave-7f01"
server_key: "ef848f4f61c757a4cd27f64a5fc06c1a"
server_key_iv: "52d7a2a65b38f14871f7d284269023f8"
server_key_version: "encryptionkey@0"
I20260812 01:04:07.794102 26942 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:04:07.797808 26963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:04:07.799264 26942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 01:04:07.799471 26942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "4524095b57924b93adb4162b2379f84d"
format_stamp: "Formatted at 2026-08-12 01:04:07 on dist-test-slave-7f01"
server_key: "ef848f4f61c757a4cd27f64a5fc06c1a"
server_key_iv: "52d7a2a65b38f14871f7d284269023f8"
server_key_version: "encryptionkey@0"
I20260812 01:04:07.799700 26942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:04:07.828660 26942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:04:07.833365 26942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:04:07.833681 26942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:04:07.854087 27052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35591 every 8 connection(s)
I20260812 01:04:07.854157 26942 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35591
I20260812 01:04:07.855573 26942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:04:07.859467 27054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:04:07.860462 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 26942
I20260812 01:04:07.860671 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:04:07.861193 22309 external_mini_cluster.cc:1428] Setting key c5aea5654bed7d8ee70ddc6075ea4630
I20260812 01:04:07.874931 27054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4524095b57924b93adb4162b2379f84d: Bootstrap starting.
I20260812 01:04:07.876995 27061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:07.863228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:54647 (local address 127.0.0.1:35591)
0812 01:04:07.864095 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:07.864110 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:07.864176 (+ 66us) server_negotiation.cc:407] Connection header received
0812 01:04:07.866508 (+ 2332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:07.866539 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:07.867110 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:07.867573 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:07.868115 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:07.870020 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:07.871411 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:07.872570 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:07.872759 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:07.874017 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:07.874028 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:07.874044 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:04:07.874781 (+ 737us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:07.874881 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:04:07.874921 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:07.875098 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:07.875418 (+ 320us) server_negotiation.cc:299] Negotiation successful
0812 01:04:07.875612 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":357,"threads_started":1}
I20260812 01:04:07.879369 27054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4524095b57924b93adb4162b2379f84d: Neither blocks nor log segments found. Creating new log.
I20260812 01:04:07.880872 27054 log.cc:824] T 00000000000000000000000000000000 P 4524095b57924b93adb4162b2379f84d: Log is configured to *not* fsync() on all Append() calls
I20260812 01:04:07.883044 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:04:07.885320 27054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4524095b57924b93adb4162b2379f84d: No bootstrap required, opened a new log
I20260812 01:04:07.895591 27061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:07.888757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:36129 (local address 127.0.0.1:35591)
0812 01:04:07.888972 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:07.888979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:07.889326 (+ 347us) server_negotiation.cc:407] Connection header received
0812 01:04:07.889467 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:07.889473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:07.889563 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:07.889683 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:07.890147 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:07.891318 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:07.892811 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:07.893542 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:07.893682 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:07.894804 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:07.894810 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:07.894816 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:04:07.894836 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:07.894906 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:04:07.894927 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:07.895057 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:07.895229 (+ 172us) server_negotiation.cc:299] Negotiation successful
0812 01:04:07.895288 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 01:04:07.896062 27054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4524095b57924b93adb4162b2379f84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4524095b57924b93adb4162b2379f84d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35591 } }
I20260812 01:04:07.896504 27054 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4524095b57924b93adb4162b2379f84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:04:07.896639 27054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4524095b57924b93adb4162b2379f84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4524095b57924b93adb4162b2379f84d, State: Initialized, Role: FOLLOWER
I20260812 01:04:07.897270 27054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4524095b57924b93adb4162b2379f84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4524095b57924b93adb4162b2379f84d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35591 } }
I20260812 01:04:07.897533 27054 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4524095b57924b93adb4162b2379f84d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:04:07.897693 27054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4524095b57924b93adb4162b2379f84d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:04:07.897866 27054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4524095b57924b93adb4162b2379f84d [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:04:07.900254 27054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4524095b57924b93adb4162b2379f84d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4524095b57924b93adb4162b2379f84d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35591 } }
I20260812 01:04:07.900779 27054 leader_election.cc:302] T 00000000000000000000000000000000 P 4524095b57924b93adb4162b2379f84d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4524095b57924b93adb4162b2379f84d; no voters:
I20260812 01:04:07.901366 27054 leader_election.cc:288] T 00000000000000000000000000000000 P 4524095b57924b93adb4162b2379f84d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:04:07.901564 27070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4524095b57924b93adb4162b2379f84d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:04:07.902304 27070 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4524095b57924b93adb4162b2379f84d [term 1 LEADER]: Becoming Leader. State: Replica: 4524095b57924b93adb4162b2379f84d, State: Running, Role: LEADER
I20260812 01:04:07.902974 27070 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4524095b57924b93adb4162b2379f84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4524095b57924b93adb4162b2379f84d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35591 } }
I20260812 01:04:07.903913 27054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4524095b57924b93adb4162b2379f84d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:04:07.906512 27072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4524095b57924b93adb4162b2379f84d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4524095b57924b93adb4162b2379f84d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4524095b57924b93adb4162b2379f84d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35591 } } }
I20260812 01:04:07.906551 27073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4524095b57924b93adb4162b2379f84d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4524095b57924b93adb4162b2379f84d. Latest consensus state: current_term: 1 leader_uuid: "4524095b57924b93adb4162b2379f84d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4524095b57924b93adb4162b2379f84d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35591 } } }
I20260812 01:04:07.906948 27072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4524095b57924b93adb4162b2379f84d [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:07.907002 27073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4524095b57924b93adb4162b2379f84d [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:07.910964 27084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:04:07.916569 27084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:04:07.926397 27084 catalog_manager.cc:1347] Generated new cluster ID: cbf34084a1d84102b32299dec9a81259
I20260812 01:04:07.926534 27084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:04:07.955900 27084 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:04:07.957851 27084 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:04:07.966122 27084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4524095b57924b93adb4162b2379f84d: Generated new TSK 0
I20260812 01:04:07.967254 27084 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:04:07.982292 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 26942
W20260812 01:04:07.983628 27045 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.49:54647
W20260812 01:04:07.984579 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 26942: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:04:07.984751 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 26942
W20260812 01:04:08.006961 26917 connection.cc:535] client connection to 127.0.0.1:35591 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35591): Connection reset by peer (error 104)
2026-08-12T01:04:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:04:08 dist-test-slave-7f01 krb5kdc[27108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:04:08 dist-test-slave-7f01 krb5kdc[27108](info): set up 2 sockets
Aug 12 01:04:08 dist-test-slave-7f01 krb5kdc[27108](info): commencing operation
krb5kdc: starting...
W20260812 01:04:09.401059 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.374s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:04:09 dist-test-slave-7f01 krb5kdc[27108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:04:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:04:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 01:04:09.777808 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36145
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:04:10.186401 27127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:04:10.186787 27127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:04:10.186969 27127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:04:10.195474 27127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:04:10.195688 27127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:04:10.195892 27127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:04:10.196005 27127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:04:10.196080 27127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:04:10.208743 27127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:04:10.210294 27127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:04:10.211727 27127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:04:10.227936 27138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:10.228088 27133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:10.228063 27134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:04:10.229024 27127 server_base.cc:1034] running on GCE node
I20260812 01:04:10.230059 27127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:04:10.231668 27127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:04:10.233014 27127 hybrid_clock.cc:648] HybridClock initialized: now 1786496650232944 us; error 76 us; skew 500 ppm
I20260812 01:04:10.233451 27127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:04:10 dist-test-slave-7f01 krb5kdc[27108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:04:10.238677 27127 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 01:04:10.240664 27127 webserver.cc:460] Webserver started at http://127.0.0.1:44437/ using document root <none> and password file <none>
I20260812 01:04:10.241428 27127 fs_manager.cc:359] Metadata directory not provided
I20260812 01:04:10.241592 27127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:04:10.241940 27127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:04:10.245283 27127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "b37f876aa33946a586b0ae25db3fc75b"
format_stamp: "Formatted at 2026-08-12 01:04:10 on dist-test-slave-7f01"
server_key: "3ce36d32e8f0d84f5de26720c2b67d11"
server_key_iv: "10d24bc88e6b812f51111af3f43f992c"
server_key_version: "encryptionkey@0"
I20260812 01:04:10.246107 27127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "b37f876aa33946a586b0ae25db3fc75b"
format_stamp: "Formatted at 2026-08-12 01:04:10 on dist-test-slave-7f01"
server_key: "3ce36d32e8f0d84f5de26720c2b67d11"
server_key_iv: "10d24bc88e6b812f51111af3f43f992c"
server_key_version: "encryptionkey@0"
I20260812 01:04:10.252436 27127 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 01:04:10.256831 27151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:04:10.258389 27127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:04:10.258749 27127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "b37f876aa33946a586b0ae25db3fc75b"
format_stamp: "Formatted at 2026-08-12 01:04:10 on dist-test-slave-7f01"
server_key: "3ce36d32e8f0d84f5de26720c2b67d11"
server_key_iv: "10d24bc88e6b812f51111af3f43f992c"
server_key_version: "encryptionkey@0"
I20260812 01:04:10.259076 27127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:04:10.267398 27127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:04:10.273893 27127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:04:10.274338 27127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:04:10.298899 27127 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36145
I20260812 01:04:10.299005 27254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36145 every 8 connection(s)
I20260812 01:04:10.300978 27127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:04:10.301302 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 27127
I20260812 01:04:10.301517 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:04:10.301975 22309 external_mini_cluster.cc:1428] Setting key 16c94718c2daf26577c84d0ae89c573b
I20260812 01:04:10.306071 27257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:04:10.312395 27261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:10.304104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:34375 (local address 127.0.0.1:36145)
0812 01:04:10.305292 (+ 1188us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:10.305313 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:10.305386 (+ 73us) server_negotiation.cc:407] Connection header received
0812 01:04:10.307703 (+ 2317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:10.307809 (+ 106us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:10.308330 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:10.308855 (+ 525us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:10.310262 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:10.310270 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:10.310286 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:04:10.310302 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:10.310391 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:04:10.310418 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:10.310819 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:10.311071 (+ 252us) server_negotiation.cc:299] Negotiation successful
0812 01:04:10.311286 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":458,"threads_started":1}
I20260812 01:04:10.316998 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:04:10.321449 27257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b37f876aa33946a586b0ae25db3fc75b: Bootstrap starting.
I20260812 01:04:10.325657 27257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b37f876aa33946a586b0ae25db3fc75b: Neither blocks nor log segments found. Creating new log.
I20260812 01:04:10.326718 27257 log.cc:824] T 00000000000000000000000000000000 P b37f876aa33946a586b0ae25db3fc75b: Log is configured to *not* fsync() on all Append() calls
I20260812 01:04:10.329989 27261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:10.323143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:60337 (local address 127.0.0.1:36145)
0812 01:04:10.323380 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:10.323391 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:10.323659 (+ 268us) server_negotiation.cc:407] Connection header received
0812 01:04:10.323876 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:10.323881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:10.323960 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:10.324085 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:10.328975 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:10.328981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:10.328986 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:04:10.329008 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:10.329069 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:04:10.329093 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:10.329247 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:10.329562 (+ 315us) server_negotiation.cc:299] Negotiation successful
0812 01:04:10.329646 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 01:04:10.331367 27257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b37f876aa33946a586b0ae25db3fc75b: No bootstrap required, opened a new log
I20260812 01:04:10.340541 27257 raft_consensus.cc:348] T 00000000000000000000000000000000 P b37f876aa33946a586b0ae25db3fc75b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37f876aa33946a586b0ae25db3fc75b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36145 } }
I20260812 01:04:10.341204 27257 raft_consensus.cc:374] T 00000000000000000000000000000000 P b37f876aa33946a586b0ae25db3fc75b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:04:10.341398 27257 raft_consensus.cc:729] T 00000000000000000000000000000000 P b37f876aa33946a586b0ae25db3fc75b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b37f876aa33946a586b0ae25db3fc75b, State: Initialized, Role: FOLLOWER
I20260812 01:04:10.342095 27257 consensus_queue.cc:260] T 00000000000000000000000000000000 P b37f876aa33946a586b0ae25db3fc75b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37f876aa33946a586b0ae25db3fc75b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36145 } }
I20260812 01:04:10.342367 27257 raft_consensus.cc:388] T 00000000000000000000000000000000 P b37f876aa33946a586b0ae25db3fc75b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:04:10.342482 27257 raft_consensus.cc:482] T 00000000000000000000000000000000 P b37f876aa33946a586b0ae25db3fc75b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:04:10.342617 27257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b37f876aa33946a586b0ae25db3fc75b [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:04:10.345465 27257 raft_consensus.cc:504] T 00000000000000000000000000000000 P b37f876aa33946a586b0ae25db3fc75b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37f876aa33946a586b0ae25db3fc75b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36145 } }
I20260812 01:04:10.346083 27257 leader_election.cc:302] T 00000000000000000000000000000000 P b37f876aa33946a586b0ae25db3fc75b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b37f876aa33946a586b0ae25db3fc75b; no voters:
I20260812 01:04:10.346735 27257 leader_election.cc:288] T 00000000000000000000000000000000 P b37f876aa33946a586b0ae25db3fc75b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:04:10.346963 27269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b37f876aa33946a586b0ae25db3fc75b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:04:10.347813 27269 raft_consensus.cc:686] T 00000000000000000000000000000000 P b37f876aa33946a586b0ae25db3fc75b [term 1 LEADER]: Becoming Leader. State: Replica: b37f876aa33946a586b0ae25db3fc75b, State: Running, Role: LEADER
I20260812 01:04:10.348579 27269 consensus_queue.cc:237] T 00000000000000000000000000000000 P b37f876aa33946a586b0ae25db3fc75b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37f876aa33946a586b0ae25db3fc75b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36145 } }
I20260812 01:04:10.349144 27257 sys_catalog.cc:564] T 00000000000000000000000000000000 P b37f876aa33946a586b0ae25db3fc75b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:04:10.352741 27270 sys_catalog.cc:455] T 00000000000000000000000000000000 P b37f876aa33946a586b0ae25db3fc75b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b37f876aa33946a586b0ae25db3fc75b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37f876aa33946a586b0ae25db3fc75b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36145 } } }
I20260812 01:04:10.353466 27270 sys_catalog.cc:458] T 00000000000000000000000000000000 P b37f876aa33946a586b0ae25db3fc75b [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:10.354538 27272 sys_catalog.cc:455] T 00000000000000000000000000000000 P b37f876aa33946a586b0ae25db3fc75b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b37f876aa33946a586b0ae25db3fc75b. Latest consensus state: current_term: 1 leader_uuid: "b37f876aa33946a586b0ae25db3fc75b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37f876aa33946a586b0ae25db3fc75b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36145 } } }
I20260812 01:04:10.355043 27272 sys_catalog.cc:458] T 00000000000000000000000000000000 P b37f876aa33946a586b0ae25db3fc75b [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:10.356405 27283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:04:10.365428 27283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:04:10.375258 27283 catalog_manager.cc:1347] Generated new cluster ID: ce86b2fd0f3a48af89a6bb06134292f2
I20260812 01:04:10.375406 27283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:04:10.389884 27283 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:04:10.390478 27283 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:04:10.405083 27283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b37f876aa33946a586b0ae25db3fc75b: Generated new TSK 0
I20260812 01:04:10.406355 27283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:04:10.422369 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 27127
W20260812 01:04:10.423609 27249 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.49:34375
W20260812 01:04:10.424540 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 27127: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:04:10.424731 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 27127
2026-08-12T01:04:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2450 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:04:10 dist-test-slave-7f01 krb5kdc[27299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:04:10 dist-test-slave-7f01 krb5kdc[27299](info): set up 2 sockets
Aug 12 01:04:10 dist-test-slave-7f01 krb5kdc[27299](info): commencing operation
krb5kdc: starting...
W20260812 01:04:11.878331 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.399s user 0.000s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:04:12 dist-test-slave-7f01 krb5kdc[27299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:04:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:04:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 01:04:12.191437 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40079
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:04:12.541838 27314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:04:12.542114 27314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:04:12.542209 27314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:04:12.549345 27314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:04:12.549456 27314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:04:12.549511 27314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:04:12.549577 27314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:04:12.549630 27314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:04:12.559196 27314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40079
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:04:12.560559 27314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:04:12.561911 27314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:04:12.573976 27331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:12.574126 27328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:12.574409 27326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:04:12.574870 27314 server_base.cc:1034] running on GCE node
I20260812 01:04:12.575716 27314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:04:12.577133 27314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:04:12.578413 27314 hybrid_clock.cc:648] HybridClock initialized: now 1786496652578358 us; error 40 us; skew 500 ppm
I20260812 01:04:12.578792 27314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:04:12 dist-test-slave-7f01 krb5kdc[27299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:04:12.582078 27314 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 01:04:12.583525 27314 webserver.cc:460] Webserver started at http://127.0.0.1:39373/ using document root <none> and password file <none>
I20260812 01:04:12.584280 27314 fs_manager.cc:359] Metadata directory not provided
I20260812 01:04:12.584436 27314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:04:12.584755 27314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:04:12.587911 27314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "8dbc34341e4c437fa3406c32a946656e"
format_stamp: "Formatted at 2026-08-12 01:04:12 on dist-test-slave-7f01"
server_key: "f7b1a5431cc2b9c6a4d3a28362119cf5"
server_key_iv: "cdbe0fafdd9030a7577d79cb7d312b5d"
server_key_version: "encryptionkey@0"
I20260812 01:04:12.588717 27314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "8dbc34341e4c437fa3406c32a946656e"
format_stamp: "Formatted at 2026-08-12 01:04:12 on dist-test-slave-7f01"
server_key: "f7b1a5431cc2b9c6a4d3a28362119cf5"
server_key_iv: "cdbe0fafdd9030a7577d79cb7d312b5d"
server_key_version: "encryptionkey@0"
I20260812 01:04:12.593918 27314 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:04:12.597249 27344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:04:12.598416 27314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 01:04:12.598610 27314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "8dbc34341e4c437fa3406c32a946656e"
format_stamp: "Formatted at 2026-08-12 01:04:12 on dist-test-slave-7f01"
server_key: "f7b1a5431cc2b9c6a4d3a28362119cf5"
server_key_iv: "cdbe0fafdd9030a7577d79cb7d312b5d"
server_key_version: "encryptionkey@0"
I20260812 01:04:12.598809 27314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:04:12.620749 27314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:04:12.624672 27314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:04:12.624970 27314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:04:12.651913 27314 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40079
I20260812 01:04:12.651929 27426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40079 every 8 connection(s)
I20260812 01:04:12.653785 27314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:04:12.657778 27430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:04:12.661356 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 27314
I20260812 01:04:12.661556 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:04:12.662127 22309 external_mini_cluster.cc:1428] Setting key dd9b8f6936e893ec8ef988a9483bb6df
I20260812 01:04:12.671200 27430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dbc34341e4c437fa3406c32a946656e: Bootstrap starting.
I20260812 01:04:12.675462 27430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dbc34341e4c437fa3406c32a946656e: Neither blocks nor log segments found. Creating new log.
I20260812 01:04:12.676806 27430 log.cc:824] T 00000000000000000000000000000000 P 8dbc34341e4c437fa3406c32a946656e: Log is configured to *not* fsync() on all Append() calls
Aug 12 01:04:12 dist-test-slave-7f01 krb5kdc[27299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 01:04:12.683153 27430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dbc34341e4c437fa3406c32a946656e: No bootstrap required, opened a new log
I20260812 01:04:12.692961 27430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8dbc34341e4c437fa3406c32a946656e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dbc34341e4c437fa3406c32a946656e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40079 } }
I20260812 01:04:12.693379 27430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8dbc34341e4c437fa3406c32a946656e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:04:12.693423 27434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:12.664321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:45161 (local address 127.0.0.1:40079)
0812 01:04:12.666361 (+ 2040us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:12.666376 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:12.666432 (+ 56us) server_negotiation.cc:407] Connection header received
0812 01:04:12.669408 (+ 2976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:12.669439 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:12.669958 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:12.670387 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:12.672298 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:12.674240 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:12.676076 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:12.677085 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:12.677897 (+ 812us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:12.682580 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:12.682593 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:12.682614 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:12.682667 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:12.687638 (+ 4971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:12.688796 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:12.688806 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:12.688817 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:12.689076 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:12.689687 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:12.689694 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:12.689700 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:12.690494 (+ 794us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:12.690875 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:12.691437 (+ 562us) server_negotiation.cc:299] Negotiation successful
0812 01:04:12.691856 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1790,"thread_start_us":1547,"threads_started":1}
I20260812 01:04:12.693548 27430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8dbc34341e4c437fa3406c32a946656e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dbc34341e4c437fa3406c32a946656e, State: Initialized, Role: FOLLOWER
I20260812 01:04:12.694211 27430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dbc34341e4c437fa3406c32a946656e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dbc34341e4c437fa3406c32a946656e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40079 } }
I20260812 01:04:12.694514 27430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8dbc34341e4c437fa3406c32a946656e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:04:12.694653 27430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8dbc34341e4c437fa3406c32a946656e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:04:12.694813 27430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8dbc34341e4c437fa3406c32a946656e [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:04:12.697837 27430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8dbc34341e4c437fa3406c32a946656e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dbc34341e4c437fa3406c32a946656e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40079 } }
I20260812 01:04:12.698400 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:04:12.698441 27430 leader_election.cc:302] T 00000000000000000000000000000000 P 8dbc34341e4c437fa3406c32a946656e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8dbc34341e4c437fa3406c32a946656e; no voters:
I20260812 01:04:12.699136 27430 leader_election.cc:288] T 00000000000000000000000000000000 P 8dbc34341e4c437fa3406c32a946656e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:04:12.699335 27442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8dbc34341e4c437fa3406c32a946656e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:04:12.706341 27442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8dbc34341e4c437fa3406c32a946656e [term 1 LEADER]: Becoming Leader. State: Replica: 8dbc34341e4c437fa3406c32a946656e, State: Running, Role: LEADER
I20260812 01:04:12.707088 27442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8dbc34341e4c437fa3406c32a946656e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dbc34341e4c437fa3406c32a946656e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40079 } }
I20260812 01:04:12.716862 27430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dbc34341e4c437fa3406c32a946656e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:04:12.720762 27445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dbc34341e4c437fa3406c32a946656e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8dbc34341e4c437fa3406c32a946656e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dbc34341e4c437fa3406c32a946656e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40079 } } }
I20260812 01:04:12.721383 27445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dbc34341e4c437fa3406c32a946656e [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:12.720708 27443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dbc34341e4c437fa3406c32a946656e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dbc34341e4c437fa3406c32a946656e. Latest consensus state: current_term: 1 leader_uuid: "8dbc34341e4c437fa3406c32a946656e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dbc34341e4c437fa3406c32a946656e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40079 } } }
I20260812 01:04:12.722326 27443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dbc34341e4c437fa3406c32a946656e [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:12.728140 27455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:04:12.738736 27434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:12.714067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:48073 (local address 127.0.0.1:40079)
0812 01:04:12.714258 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:12.714265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:12.715078 (+ 813us) server_negotiation.cc:407] Connection header received
0812 01:04:12.715354 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:12.715359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:12.715443 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:12.715515 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:12.718117 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:12.719381 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:12.723494 (+ 4113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:12.724350 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:12.724469 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:12.728698 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:12.728704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:12.728709 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:12.728750 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:12.733371 (+ 4621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:12.734624 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:12.734633 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:12.734639 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:12.734829 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:12.735535 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:12.735543 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:12.735549 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:12.736170 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:12.737694 (+ 1524us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:12.737938 (+ 244us) server_negotiation.cc:299] Negotiation successful
0812 01:04:12.738196 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 01:04:12.744458 27455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:04:12.769677 27455 catalog_manager.cc:1347] Generated new cluster ID: 9c32c745b38747498095a806608fa657
I20260812 01:04:12.769804 27455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:04:12.796176 27455 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:04:12.798146 27455 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:04:12.806699 27455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8dbc34341e4c437fa3406c32a946656e: Generated new TSK 0
I20260812 01:04:12.807703 27455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:04:12.818488 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 27314
I20260812 01:04:12.819856 27421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:04:12.919267 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 27314
W20260812 01:04:12.943480 27295 connection.cc:535] client connection to 127.0.0.1:40079 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40079): Connection reset by peer (error 104)
2026-08-12T01:04:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2490 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:04:13 dist-test-slave-7f01 krb5kdc[27479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:04:13 dist-test-slave-7f01 krb5kdc[27479](info): set up 2 sockets
Aug 12 01:04:13 dist-test-slave-7f01 krb5kdc[27479](info): commencing operation
krb5kdc: starting...
W20260812 01:04:14.304368 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.336s user 0.006s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:04:14 dist-test-slave-7f01 krb5kdc[27479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:04:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:04:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 01:04:14.606529 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42643
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:04:14.950843 27496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:04:14.951143 27496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:04:14.951321 27496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:04:14.958671 27496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:04:14.958786 27496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:04:14.958904 27496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:04:14.959041 27496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:04:14.959122 27496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:04:14.968989 27496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42643
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:04:14.970345 27496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:04:14.971664 27496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:04:14.983673 27511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:14.983759 27506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:14.983951 27505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:04:14.985283 27496 server_base.cc:1034] running on GCE node
I20260812 01:04:14.986050 27496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:04:14.987393 27496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:04:14.988629 27496 hybrid_clock.cc:648] HybridClock initialized: now 1786496654988601 us; error 46 us; skew 500 ppm
I20260812 01:04:14.988986 27496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:04:14 dist-test-slave-7f01 krb5kdc[27479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:04:14.992297 27496 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 01:04:14.993863 27496 webserver.cc:460] Webserver started at http://127.0.0.1:37919/ using document root <none> and password file <none>
I20260812 01:04:14.994567 27496 fs_manager.cc:359] Metadata directory not provided
I20260812 01:04:14.994719 27496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:04:14.995049 27496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:04:14.998175 27496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "cfa6bead95614a7cb81419a2f83d1e35"
format_stamp: "Formatted at 2026-08-12 01:04:14 on dist-test-slave-7f01"
server_key: "106a5c85e46596b02c72d9ae1bf2ec64"
server_key_iv: "413a8c3a3c7b164b6f02db6e592a5704"
server_key_version: "encryptionkey@0"
I20260812 01:04:14.998970 27496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "cfa6bead95614a7cb81419a2f83d1e35"
format_stamp: "Formatted at 2026-08-12 01:04:14 on dist-test-slave-7f01"
server_key: "106a5c85e46596b02c72d9ae1bf2ec64"
server_key_iv: "413a8c3a3c7b164b6f02db6e592a5704"
server_key_version: "encryptionkey@0"
I20260812 01:04:15.004030 27496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 01:04:15.007305 27523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:04:15.008598 27496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:04:15.008785 27496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "cfa6bead95614a7cb81419a2f83d1e35"
format_stamp: "Formatted at 2026-08-12 01:04:14 on dist-test-slave-7f01"
server_key: "106a5c85e46596b02c72d9ae1bf2ec64"
server_key_iv: "413a8c3a3c7b164b6f02db6e592a5704"
server_key_version: "encryptionkey@0"
I20260812 01:04:15.008994 27496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:04:15.023387 27496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:04:15.027534 27496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:04:15.027884 27496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:04:15.046362 27496 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42643
I20260812 01:04:15.046409 27600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42643 every 8 connection(s)
I20260812 01:04:15.047734 27496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:04:15.051347 27602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:04:15.056072 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 27496
I20260812 01:04:15.056274 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:04:15.056743 22309 external_mini_cluster.cc:1428] Setting key 3a4076afce4fbc9a0658f38431d8c64e
I20260812 01:04:15.062273 27602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfa6bead95614a7cb81419a2f83d1e35: Bootstrap starting.
I20260812 01:04:15.065903 27602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfa6bead95614a7cb81419a2f83d1e35: Neither blocks nor log segments found. Creating new log.
I20260812 01:04:15.067154 27602 log.cc:824] T 00000000000000000000000000000000 P cfa6bead95614a7cb81419a2f83d1e35: Log is configured to *not* fsync() on all Append() calls
I20260812 01:04:15.070449 27602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfa6bead95614a7cb81419a2f83d1e35: No bootstrap required, opened a new log
I20260812 01:04:15.070638 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:15.058477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:49105 (local address 127.0.0.1:42643)
0812 01:04:15.059280 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:15.059291 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:15.059331 (+ 40us) server_negotiation.cc:407] Connection header received
0812 01:04:15.061122 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:15.061143 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:15.061496 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:15.061781 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:15.062417 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:15.063731 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:15.065713 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:15.066448 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:15.066574 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:15.068064 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:15.068071 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:15.068083 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:04:15.068678 (+ 595us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:15.068772 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:04:15.068813 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:15.068983 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:15.069241 (+ 258us) server_negotiation.cc:299] Negotiation successful
0812 01:04:15.069406 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":368,"threads_started":1}
I20260812 01:04:15.074460 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:04:15.078842 27602 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfa6bead95614a7cb81419a2f83d1e35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa6bead95614a7cb81419a2f83d1e35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42643 } }
I20260812 01:04:15.079365 27602 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfa6bead95614a7cb81419a2f83d1e35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:04:15.079479 27602 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfa6bead95614a7cb81419a2f83d1e35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfa6bead95614a7cb81419a2f83d1e35, State: Initialized, Role: FOLLOWER
I20260812 01:04:15.080125 27602 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfa6bead95614a7cb81419a2f83d1e35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa6bead95614a7cb81419a2f83d1e35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42643 } }
I20260812 01:04:15.080394 27602 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfa6bead95614a7cb81419a2f83d1e35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:04:15.080544 27602 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfa6bead95614a7cb81419a2f83d1e35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:04:15.080729 27602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfa6bead95614a7cb81419a2f83d1e35 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:04:15.083237 27602 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfa6bead95614a7cb81419a2f83d1e35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa6bead95614a7cb81419a2f83d1e35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42643 } }
I20260812 01:04:15.083721 27602 leader_election.cc:302] T 00000000000000000000000000000000 P cfa6bead95614a7cb81419a2f83d1e35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfa6bead95614a7cb81419a2f83d1e35; no voters:
I20260812 01:04:15.084512 27602 leader_election.cc:288] T 00000000000000000000000000000000 P cfa6bead95614a7cb81419a2f83d1e35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:04:15.084635 27621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfa6bead95614a7cb81419a2f83d1e35 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:04:15.086262 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:15.078996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:33161 (local address 127.0.0.1:42643)
0812 01:04:15.079171 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:15.079178 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:15.079764 (+ 586us) server_negotiation.cc:407] Connection header received
0812 01:04:15.079976 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:15.079981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:15.080052 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:15.080163 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:15.080735 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:15.081585 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:15.082854 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:15.083326 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:15.083460 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:15.084870 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:15.084876 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:15.084884 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:04:15.084914 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:15.084975 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:04:15.085000 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:15.085142 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:15.085718 (+ 576us) server_negotiation.cc:299] Negotiation successful
0812 01:04:15.085787 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 01:04:15.086658 27602 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfa6bead95614a7cb81419a2f83d1e35 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:04:15.088341 27621 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfa6bead95614a7cb81419a2f83d1e35 [term 1 LEADER]: Becoming Leader. State: Replica: cfa6bead95614a7cb81419a2f83d1e35, State: Running, Role: LEADER
I20260812 01:04:15.088919 27621 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfa6bead95614a7cb81419a2f83d1e35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa6bead95614a7cb81419a2f83d1e35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42643 } }
I20260812 01:04:15.091843 27625 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfa6bead95614a7cb81419a2f83d1e35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfa6bead95614a7cb81419a2f83d1e35. Latest consensus state: current_term: 1 leader_uuid: "cfa6bead95614a7cb81419a2f83d1e35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa6bead95614a7cb81419a2f83d1e35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42643 } } }
I20260812 01:04:15.091817 27623 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfa6bead95614a7cb81419a2f83d1e35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfa6bead95614a7cb81419a2f83d1e35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa6bead95614a7cb81419a2f83d1e35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42643 } } }
I20260812 01:04:15.092281 27623 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfa6bead95614a7cb81419a2f83d1e35 [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:15.092281 27625 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfa6bead95614a7cb81419a2f83d1e35 [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:15.093185 27631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:04:15.098410 27631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:04:15.106297 27631 catalog_manager.cc:1347] Generated new cluster ID: 9a4da12feeae47678d2bc7419810696c
I20260812 01:04:15.106402 27631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:04:15.124670 27631 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:04:15.125810 27631 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:04:15.134582 27631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfa6bead95614a7cb81419a2f83d1e35: Generated new TSK 0
I20260812 01:04:15.135278 27631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:04:15.147324 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 27496
W20260812 01:04:15.149549 27596 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.49:49105
W20260812 01:04:15.150349 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 27496: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:04:15.150511 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 27496
W20260812 01:04:15.174196 27474 connection.cc:535] client connection to 127.0.0.1:42643 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42643): Connection reset by peer (error 104)
2026-08-12T01:04:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2225 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:04:15 dist-test-slave-7f01 krb5kdc[27650](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:04:15 dist-test-slave-7f01 krb5kdc[27650](info): set up 2 sockets
Aug 12 01:04:15 dist-test-slave-7f01 krb5kdc[27650](info): commencing operation
krb5kdc: starting...
W20260812 01:04:16.518774 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.325s user 0.003s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:04:16 dist-test-slave-7f01 krb5kdc[27650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:04:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:04:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 01:04:16.835192 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37059
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:04:17.184386 27681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:04:17.184670 27681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:04:17.184868 27681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:04:17.192049 27681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:04:17.192189 27681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:04:17.192312 27681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:04:17.192435 27681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:04:17.192520 27681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:04:17.202446 27681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37059
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:04:17.203722 27681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:04:17.205062 27681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:04:17.217237 27689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:17.217204 27686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:17.217456 27691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:04:17.218178 27681 server_base.cc:1034] running on GCE node
I20260812 01:04:17.219060 27681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:04:17.220507 27681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:04:17.221750 27681 hybrid_clock.cc:648] HybridClock initialized: now 1786496657221713 us; error 46 us; skew 500 ppm
I20260812 01:04:17.222098 27681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:04:17 dist-test-slave-7f01 krb5kdc[27650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:04:17.229696 27681 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 01:04:17.231184 27681 webserver.cc:460] Webserver started at http://127.0.0.1:39263/ using document root <none> and password file <none>
I20260812 01:04:17.231971 27681 fs_manager.cc:359] Metadata directory not provided
I20260812 01:04:17.232129 27681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:04:17.232453 27681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:04:17.235477 27681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "605a584c06a5498c95250e65e9a2f03d"
format_stamp: "Formatted at 2026-08-12 01:04:17 on dist-test-slave-7f01"
server_key: "8de64bd031867e3d61f67fef24a3e1bd"
server_key_iv: "a643659b82ba35ddc1bb992c1695e507"
server_key_version: "encryptionkey@0"
I20260812 01:04:17.236352 27681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "605a584c06a5498c95250e65e9a2f03d"
format_stamp: "Formatted at 2026-08-12 01:04:17 on dist-test-slave-7f01"
server_key: "8de64bd031867e3d61f67fef24a3e1bd"
server_key_iv: "a643659b82ba35ddc1bb992c1695e507"
server_key_version: "encryptionkey@0"
I20260812 01:04:17.241555 27681 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 01:04:17.244999 27699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:04:17.246176 27681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:04:17.246356 27681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "605a584c06a5498c95250e65e9a2f03d"
format_stamp: "Formatted at 2026-08-12 01:04:17 on dist-test-slave-7f01"
server_key: "8de64bd031867e3d61f67fef24a3e1bd"
server_key_iv: "a643659b82ba35ddc1bb992c1695e507"
server_key_version: "encryptionkey@0"
I20260812 01:04:17.246570 27681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:04:17.252516 27681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:04:17.256423 27681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:04:17.256732 27681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:04:17.275718 27681 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37059
I20260812 01:04:17.275784 27788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37059 every 8 connection(s)
I20260812 01:04:17.277170 27681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:04:17.280726 27789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:04:17.284436 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 27681
I20260812 01:04:17.284636 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:04:17.285208 22309 external_mini_cluster.cc:1428] Setting key a7cc61fa1bac54174bdc55c50e89cb97
I20260812 01:04:17.292753 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:17.286999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:60839 (local address 127.0.0.1:37059)
0812 01:04:17.287810 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:17.287822 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:17.287863 (+ 41us) server_negotiation.cc:407] Connection header received
0812 01:04:17.289470 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:17.289498 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:17.289822 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:17.290135 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:17.291201 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:17.291207 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:17.291222 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:04:17.291238 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:17.291294 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:04:17.291315 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:17.291501 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:17.291595 (+ 94us) server_negotiation.cc:299] Negotiation successful
0812 01:04:17.291822 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":330,"threads_started":1}
I20260812 01:04:17.293481 27789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 605a584c06a5498c95250e65e9a2f03d: Bootstrap starting.
I20260812 01:04:17.296470 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:04:17.296875 27789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 605a584c06a5498c95250e65e9a2f03d: Neither blocks nor log segments found. Creating new log.
I20260812 01:04:17.297966 27789 log.cc:824] T 00000000000000000000000000000000 P 605a584c06a5498c95250e65e9a2f03d: Log is configured to *not* fsync() on all Append() calls
I20260812 01:04:17.300918 27789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 605a584c06a5498c95250e65e9a2f03d: No bootstrap required, opened a new log
I20260812 01:04:17.304119 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:17.301444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:50333 (local address 127.0.0.1:37059)
0812 01:04:17.301608 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:17.301612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:17.302089 (+ 477us) server_negotiation.cc:407] Connection header received
0812 01:04:17.302280 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:17.302287 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:17.302351 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:17.302460 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:17.303408 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:17.303414 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:17.303418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:04:17.303436 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:17.303488 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:04:17.303508 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:17.303658 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:17.303774 (+ 116us) server_negotiation.cc:299] Negotiation successful
0812 01:04:17.303840 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 01:04:17.309484 27789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 605a584c06a5498c95250e65e9a2f03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605a584c06a5498c95250e65e9a2f03d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37059 } }
I20260812 01:04:17.309854 27789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 605a584c06a5498c95250e65e9a2f03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:04:17.309981 27789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 605a584c06a5498c95250e65e9a2f03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 605a584c06a5498c95250e65e9a2f03d, State: Initialized, Role: FOLLOWER
I20260812 01:04:17.310534 27789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 605a584c06a5498c95250e65e9a2f03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605a584c06a5498c95250e65e9a2f03d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37059 } }
I20260812 01:04:17.310771 27789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 605a584c06a5498c95250e65e9a2f03d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:04:17.310943 27789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 605a584c06a5498c95250e65e9a2f03d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:04:17.311105 27789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 605a584c06a5498c95250e65e9a2f03d [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:04:17.313673 27789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 605a584c06a5498c95250e65e9a2f03d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605a584c06a5498c95250e65e9a2f03d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37059 } }
I20260812 01:04:17.314183 27789 leader_election.cc:302] T 00000000000000000000000000000000 P 605a584c06a5498c95250e65e9a2f03d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 605a584c06a5498c95250e65e9a2f03d; no voters:
I20260812 01:04:17.314785 27789 leader_election.cc:288] T 00000000000000000000000000000000 P 605a584c06a5498c95250e65e9a2f03d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:04:17.314922 27802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 605a584c06a5498c95250e65e9a2f03d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:04:17.315690 27802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 605a584c06a5498c95250e65e9a2f03d [term 1 LEADER]: Becoming Leader. State: Replica: 605a584c06a5498c95250e65e9a2f03d, State: Running, Role: LEADER
I20260812 01:04:17.316218 27802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 605a584c06a5498c95250e65e9a2f03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605a584c06a5498c95250e65e9a2f03d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37059 } }
I20260812 01:04:17.316973 27789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 605a584c06a5498c95250e65e9a2f03d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:04:17.319581 27805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 605a584c06a5498c95250e65e9a2f03d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 605a584c06a5498c95250e65e9a2f03d. Latest consensus state: current_term: 1 leader_uuid: "605a584c06a5498c95250e65e9a2f03d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605a584c06a5498c95250e65e9a2f03d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37059 } } }
I20260812 01:04:17.320117 27805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 605a584c06a5498c95250e65e9a2f03d [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:17.321187 27804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 605a584c06a5498c95250e65e9a2f03d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "605a584c06a5498c95250e65e9a2f03d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605a584c06a5498c95250e65e9a2f03d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37059 } } }
I20260812 01:04:17.322911 27804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 605a584c06a5498c95250e65e9a2f03d [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:17.323261 27810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:04:17.328313 27810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:04:17.337522 27810 catalog_manager.cc:1347] Generated new cluster ID: 292fb910ccc44b65ac8de80bb301cd62
I20260812 01:04:17.337648 27810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:04:17.348213 27810 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:04:17.348619 27810 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:04:17.359632 27810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 605a584c06a5498c95250e65e9a2f03d: Generated new TSK 0
I20260812 01:04:17.360432 27810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:04:17.365991 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 27681
W20260812 01:04:17.367213 27779 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.49:60839
W20260812 01:04:17.367887 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 27681: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:04:17.368052 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 27681
2026-08-12T01:04:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2226 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:04:17 dist-test-slave-7f01 krb5kdc[27836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:04:17 dist-test-slave-7f01 krb5kdc[27836](info): set up 2 sockets
Aug 12 01:04:17 dist-test-slave-7f01 krb5kdc[27836](info): commencing operation
krb5kdc: starting...
W20260812 01:04:18.775139 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.355s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:04:18 dist-test-slave-7f01 krb5kdc[27836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:04:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:04:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 01:04:19.099375 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:35099
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:46197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:35099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:04:19.445480 27853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:04:19.445837 27853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:04:19.446025 27853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:04:19.453310 27853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:04:19.453470 27853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:04:19.453583 27853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:04:19.453698 27853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:04:19.453789 27853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:04:19.463522 27853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:46197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:35099
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:35099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:04:19.464924 27853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:04:19.466238 27853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:04:19.478240 27861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:19.478200 27868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:19.478283 27865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:04:19.479465 27853 server_base.cc:1034] running on GCE node
I20260812 01:04:19.480381 27853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:04:19.481737 27853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:04:19.482941 27853 hybrid_clock.cc:648] HybridClock initialized: now 1786496659482886 us; error 47 us; skew 500 ppm
I20260812 01:04:19.483290 27853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:04:19 dist-test-slave-7f01 krb5kdc[27836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496659, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:04:19.486509 27853 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 01:04:19.488138 27853 webserver.cc:460] Webserver started at http://127.21.201.126:35113/ using document root <none> and password file <none>
I20260812 01:04:19.488826 27853 fs_manager.cc:359] Metadata directory not provided
I20260812 01:04:19.488987 27853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:04:19.489305 27853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:04:19.492409 27853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "4522be16355e462b9075f20d2dcfa7f2"
format_stamp: "Formatted at 2026-08-12 01:04:19 on dist-test-slave-7f01"
server_key: "d174cc1426b53f819013f168883a54d1"
server_key_iv: "93796f3ba02e53f4d715d8aef524fd91"
server_key_version: "encryptionkey@0"
I20260812 01:04:19.493237 27853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "4522be16355e462b9075f20d2dcfa7f2"
format_stamp: "Formatted at 2026-08-12 01:04:19 on dist-test-slave-7f01"
server_key: "d174cc1426b53f819013f168883a54d1"
server_key_iv: "93796f3ba02e53f4d715d8aef524fd91"
server_key_version: "encryptionkey@0"
I20260812 01:04:19.498432 27853 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 01:04:19.501945 27876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:04:19.503257 27853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 01:04:19.503463 27853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "4522be16355e462b9075f20d2dcfa7f2"
format_stamp: "Formatted at 2026-08-12 01:04:19 on dist-test-slave-7f01"
server_key: "d174cc1426b53f819013f168883a54d1"
server_key_iv: "93796f3ba02e53f4d715d8aef524fd91"
server_key_version: "encryptionkey@0"
I20260812 01:04:19.503679 27853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:04:19.519945 27853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:04:19.524021 27853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:04:19.524331 27853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:04:19.543280 27853 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:35099
I20260812 01:04:19.543304 27967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:35099 every 8 connection(s)
I20260812 01:04:19.544729 27853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:04:19.548388 27968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:04:19.549376 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 27853
I20260812 01:04:19.549590 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:04:19.550220 22309 external_mini_cluster.cc:1428] Setting key fb5ee63e0c9f15abba39db42a2107efb
I20260812 01:04:19.562623 27968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4522be16355e462b9075f20d2dcfa7f2: Bootstrap starting.
Aug 12 01:04:19 dist-test-slave-7f01 krb5kdc[27836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:04:19.566413 27968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4522be16355e462b9075f20d2dcfa7f2: Neither blocks nor log segments found. Creating new log.
I20260812 01:04:19.567905 27968 log.cc:824] T 00000000000000000000000000000000 P 4522be16355e462b9075f20d2dcfa7f2: Log is configured to *not* fsync() on all Append() calls
I20260812 01:04:19.571331 27968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4522be16355e462b9075f20d2dcfa7f2: No bootstrap required, opened a new log
I20260812 01:04:19.575482 27974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:19.552087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55645 (local address 127.21.201.126:35099)
0812 01:04:19.552916 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:19.552933 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:19.552972 (+ 39us) server_negotiation.cc:407] Connection header received
0812 01:04:19.554621 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:19.554645 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:19.555023 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:19.555342 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:19.557319 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:19.558615 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:19.560489 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:19.561146 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:19.561283 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:19.566419 (+ 5136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:19.566432 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:19.566452 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:19.566511 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:19.570561 (+ 4050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:19.571433 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:19.571441 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:19.571449 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:19.571683 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:19.572226 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:19.572231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:19.572235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:19.572928 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:19.573293 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:19.573754 (+ 461us) server_negotiation.cc:299] Negotiation successful
0812 01:04:19.574116 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":383,"threads_started":1}
I20260812 01:04:19.579627 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:04:19.581311 27968 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4522be16355e462b9075f20d2dcfa7f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4522be16355e462b9075f20d2dcfa7f2" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 35099 } }
I20260812 01:04:19.581686 27968 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4522be16355e462b9075f20d2dcfa7f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:04:19.581794 27968 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4522be16355e462b9075f20d2dcfa7f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4522be16355e462b9075f20d2dcfa7f2, State: Initialized, Role: FOLLOWER
I20260812 01:04:19.582401 27968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4522be16355e462b9075f20d2dcfa7f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4522be16355e462b9075f20d2dcfa7f2" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 35099 } }
I20260812 01:04:19.582744 27968 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4522be16355e462b9075f20d2dcfa7f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:04:19.582913 27968 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4522be16355e462b9075f20d2dcfa7f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:04:19.583112 27968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4522be16355e462b9075f20d2dcfa7f2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:04:19.585896 27968 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4522be16355e462b9075f20d2dcfa7f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4522be16355e462b9075f20d2dcfa7f2" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 35099 } }
I20260812 01:04:19.586460 27968 leader_election.cc:302] T 00000000000000000000000000000000 P 4522be16355e462b9075f20d2dcfa7f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4522be16355e462b9075f20d2dcfa7f2; no voters:
I20260812 01:04:19.587129 27968 leader_election.cc:288] T 00000000000000000000000000000000 P 4522be16355e462b9075f20d2dcfa7f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:04:19.587246 27983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4522be16355e462b9075f20d2dcfa7f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:04:19.589777 27968 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4522be16355e462b9075f20d2dcfa7f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:04:19.590467 27983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4522be16355e462b9075f20d2dcfa7f2 [term 1 LEADER]: Becoming Leader. State: Replica: 4522be16355e462b9075f20d2dcfa7f2, State: Running, Role: LEADER
I20260812 01:04:19.591179 27983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4522be16355e462b9075f20d2dcfa7f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4522be16355e462b9075f20d2dcfa7f2" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 35099 } }
I20260812 01:04:19.594570 27986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4522be16355e462b9075f20d2dcfa7f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4522be16355e462b9075f20d2dcfa7f2. Latest consensus state: current_term: 1 leader_uuid: "4522be16355e462b9075f20d2dcfa7f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4522be16355e462b9075f20d2dcfa7f2" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 35099 } } }
I20260812 01:04:19.595901 27986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4522be16355e462b9075f20d2dcfa7f2 [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:19.600332 27984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4522be16355e462b9075f20d2dcfa7f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4522be16355e462b9075f20d2dcfa7f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4522be16355e462b9075f20d2dcfa7f2" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 35099 } } }
I20260812 01:04:19.600792 27984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4522be16355e462b9075f20d2dcfa7f2 [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:19.602205 28002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:04:19.603900 27974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:19.584999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59145 (local address 127.21.201.126:35099)
0812 01:04:19.585191 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:19.585197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:19.585640 (+ 443us) server_negotiation.cc:407] Connection header received
0812 01:04:19.585784 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:19.585789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:19.585876 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:19.585997 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:19.587798 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:19.588827 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:19.590205 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:19.590871 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:19.591037 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:19.594044 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:19.594050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:19.594054 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:19.594088 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:19.599016 (+ 4928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:19.600603 (+ 1587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:19.600608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:19.600611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:19.600777 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:19.601596 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:19.601605 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:19.601611 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:19.602257 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:19.602628 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:19.603270 (+ 642us) server_negotiation.cc:299] Negotiation successful
0812 01:04:19.603465 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 01:04:19.608897 28002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:04:19.618242 28002 catalog_manager.cc:1347] Generated new cluster ID: 7e5563300ea54e81827c0860151ed730
I20260812 01:04:19.618369 28002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:04:19.632982 28002 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:04:19.634210 28002 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:04:19.640540 28002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4522be16355e462b9075f20d2dcfa7f2: Generated new TSK 0
I20260812 01:04:19.641566 28002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:04:19.650473 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 27853
I20260812 01:04:19.652262 27961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:04:19.744450 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 27853
W20260812 01:04:19.771335 27834 connection.cc:535] client connection to 127.21.201.126:35099 recv error: Network error: failed to read from TLS socket (remote: 127.21.201.126:35099): Connection reset by peer (error 104)
2026-08-12T01:04:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:04:19 dist-test-slave-7f01 krb5kdc[28027](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:04:19 dist-test-slave-7f01 krb5kdc[28027](info): set up 2 sockets
Aug 12 01:04:19 dist-test-slave-7f01 krb5kdc[28027](info): commencing operation
krb5kdc: starting...
W20260812 01:04:21.122537 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.332s user 0.002s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:04:21 dist-test-slave-7f01 krb5kdc[28027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:04:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:04:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 01:04:21.439495 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:46513
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:36729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:46513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:04:21.789592 28046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:04:21.789891 28046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:04:21.789991 28046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:04:21.797242 28046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:04:21.797353 28046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:04:21.797407 28046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:04:21.797465 28046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:04:21.797503 28046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:04:21.807336 28046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:36729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:46513
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:46513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:04:21.808732 28046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:04:21.810022 28046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:04:21.822199 28055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:21.822269 28056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:21.822233 28061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:04:21.823323 28046 server_base.cc:1034] running on GCE node
I20260812 01:04:21.824146 28046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:04:21.825472 28046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:04:21.826674 28046 hybrid_clock.cc:648] HybridClock initialized: now 1786496661826633 us; error 30 us; skew 500 ppm
I20260812 01:04:21.827044 28046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:04:21 dist-test-slave-7f01 krb5kdc[28027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496661, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:04:21.830240 28046 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 01:04:21.831801 28046 webserver.cc:460] Webserver started at http://127.21.201.126:34253/ using document root <none> and password file <none>
I20260812 01:04:21.832448 28046 fs_manager.cc:359] Metadata directory not provided
I20260812 01:04:21.832556 28046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:04:21.832805 28046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:04:21.835810 28046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "be793edf17354555bb7c98c32b42fbff"
format_stamp: "Formatted at 2026-08-12 01:04:21 on dist-test-slave-7f01"
server_key: "0efd16e9349e1d7ecdc5393295b13bc4"
server_key_iv: "43b3311f5dbcada89fee4d97e6ee3983"
server_key_version: "encryptionkey@0"
I20260812 01:04:21.836560 28046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "be793edf17354555bb7c98c32b42fbff"
format_stamp: "Formatted at 2026-08-12 01:04:21 on dist-test-slave-7f01"
server_key: "0efd16e9349e1d7ecdc5393295b13bc4"
server_key_iv: "43b3311f5dbcada89fee4d97e6ee3983"
server_key_version: "encryptionkey@0"
I20260812 01:04:21.841549 28046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260812 01:04:21.844982 28069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:04:21.846263 28046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 01:04:21.846462 28046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "be793edf17354555bb7c98c32b42fbff"
format_stamp: "Formatted at 2026-08-12 01:04:21 on dist-test-slave-7f01"
server_key: "0efd16e9349e1d7ecdc5393295b13bc4"
server_key_iv: "43b3311f5dbcada89fee4d97e6ee3983"
server_key_version: "encryptionkey@0"
I20260812 01:04:21.846650 28046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:04:21.870402 28046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:04:21.874758 28046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:04:21.875082 28046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:04:21.899494 28046 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:46513
I20260812 01:04:21.899519 28152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:46513 every 8 connection(s)
I20260812 01:04:21.901183 28046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:04:21.905251 28156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:04:21.910363 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 28046
I20260812 01:04:21.910575 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:04:21.911139 22309 external_mini_cluster.cc:1428] Setting key 24d73cc31eb43754e7ef1318bf9b11ee
I20260812 01:04:21.917515 28156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be793edf17354555bb7c98c32b42fbff: Bootstrap starting.
I20260812 01:04:21.921145 28156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be793edf17354555bb7c98c32b42fbff: Neither blocks nor log segments found. Creating new log.
I20260812 01:04:21.922497 28156 log.cc:824] T 00000000000000000000000000000000 P be793edf17354555bb7c98c32b42fbff: Log is configured to *not* fsync() on all Append() calls
I20260812 01:04:21.925544 28160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:21.913058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51201 (local address 127.21.201.126:46513)
0812 01:04:21.913887 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:21.913899 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:21.913942 (+ 43us) server_negotiation.cc:407] Connection header received
0812 01:04:21.915982 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:21.916004 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:21.916368 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:21.916672 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:21.917330 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:21.918744 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:21.920726 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:21.921470 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:21.921625 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:21.922963 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:21.922971 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:21.923014 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:04:21.923867 (+ 853us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:21.923973 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:04:21.924011 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:21.924131 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:21.924415 (+ 284us) server_negotiation.cc:299] Negotiation successful
0812 01:04:21.924623 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":389,"threads_started":1}
I20260812 01:04:21.925688 28156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be793edf17354555bb7c98c32b42fbff: No bootstrap required, opened a new log
I20260812 01:04:21.929343 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:04:21.935845 28156 raft_consensus.cc:348] T 00000000000000000000000000000000 P be793edf17354555bb7c98c32b42fbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be793edf17354555bb7c98c32b42fbff" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46513 } }
I20260812 01:04:21.936383 28156 raft_consensus.cc:374] T 00000000000000000000000000000000 P be793edf17354555bb7c98c32b42fbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:04:21.936488 28156 raft_consensus.cc:729] T 00000000000000000000000000000000 P be793edf17354555bb7c98c32b42fbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be793edf17354555bb7c98c32b42fbff, State: Initialized, Role: FOLLOWER
I20260812 01:04:21.937215 28156 consensus_queue.cc:260] T 00000000000000000000000000000000 P be793edf17354555bb7c98c32b42fbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be793edf17354555bb7c98c32b42fbff" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46513 } }
I20260812 01:04:21.937530 28156 raft_consensus.cc:388] T 00000000000000000000000000000000 P be793edf17354555bb7c98c32b42fbff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:04:21.937675 28156 raft_consensus.cc:482] T 00000000000000000000000000000000 P be793edf17354555bb7c98c32b42fbff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:04:21.937855 28156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be793edf17354555bb7c98c32b42fbff [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:04:21.940635 28156 raft_consensus.cc:504] T 00000000000000000000000000000000 P be793edf17354555bb7c98c32b42fbff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be793edf17354555bb7c98c32b42fbff" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46513 } }
I20260812 01:04:21.941130 28156 leader_election.cc:302] T 00000000000000000000000000000000 P be793edf17354555bb7c98c32b42fbff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be793edf17354555bb7c98c32b42fbff; no voters:
I20260812 01:04:21.941875 28156 leader_election.cc:288] T 00000000000000000000000000000000 P be793edf17354555bb7c98c32b42fbff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:04:21.942037 28170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be793edf17354555bb7c98c32b42fbff [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:04:21.942334 28160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:21.934829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37677 (local address 127.21.201.126:46513)
0812 01:04:21.935001 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:21.935007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:21.935583 (+ 576us) server_negotiation.cc:407] Connection header received
0812 01:04:21.935900 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:21.935906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:21.935990 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:21.936106 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:21.936809 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:21.937980 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:21.939315 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:21.940069 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:21.940190 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:21.941337 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:21.941343 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:21.941348 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:04:21.941368 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:21.941434 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:04:21.941456 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:21.941589 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:21.941853 (+ 264us) server_negotiation.cc:299] Negotiation successful
0812 01:04:21.941922 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 01:04:21.944142 28170 raft_consensus.cc:686] T 00000000000000000000000000000000 P be793edf17354555bb7c98c32b42fbff [term 1 LEADER]: Becoming Leader. State: Replica: be793edf17354555bb7c98c32b42fbff, State: Running, Role: LEADER
I20260812 01:04:21.944253 28156 sys_catalog.cc:564] T 00000000000000000000000000000000 P be793edf17354555bb7c98c32b42fbff [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:04:21.944713 28170 consensus_queue.cc:237] T 00000000000000000000000000000000 P be793edf17354555bb7c98c32b42fbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be793edf17354555bb7c98c32b42fbff" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46513 } }
I20260812 01:04:21.947261 28171 sys_catalog.cc:455] T 00000000000000000000000000000000 P be793edf17354555bb7c98c32b42fbff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be793edf17354555bb7c98c32b42fbff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be793edf17354555bb7c98c32b42fbff" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46513 } } }
I20260812 01:04:21.947376 28172 sys_catalog.cc:455] T 00000000000000000000000000000000 P be793edf17354555bb7c98c32b42fbff [sys.catalog]: SysCatalogTable state changed. Reason: New leader be793edf17354555bb7c98c32b42fbff. Latest consensus state: current_term: 1 leader_uuid: "be793edf17354555bb7c98c32b42fbff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be793edf17354555bb7c98c32b42fbff" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46513 } } }
I20260812 01:04:21.947685 28171 sys_catalog.cc:458] T 00000000000000000000000000000000 P be793edf17354555bb7c98c32b42fbff [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:21.947702 28172 sys_catalog.cc:458] T 00000000000000000000000000000000 P be793edf17354555bb7c98c32b42fbff [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:21.949450 28181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:04:21.954223 28181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:04:21.963313 28181 catalog_manager.cc:1347] Generated new cluster ID: be4459f09e184f9cb0d451457a54681a
I20260812 01:04:21.963419 28181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:04:21.994758 28181 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:04:21.996136 28181 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:04:22.006994 28181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be793edf17354555bb7c98c32b42fbff: Generated new TSK 0
I20260812 01:04:22.007884 28181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:04:22.017966 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 28046
W20260812 01:04:22.019199 28145 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51201
W20260812 01:04:22.020035 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 28046: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:04:22.020188 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 28046
W20260812 01:04:22.046648 28024 connection.cc:535] client connection to 127.21.201.126:46513 recv error: Network error: failed to read from TLS socket (remote: 127.21.201.126:46513): Connection reset by peer (error 104)
2026-08-12T01:04:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2275 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:04:22 dist-test-slave-7f01 krb5kdc[28206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:04:22 dist-test-slave-7f01 krb5kdc[28206](info): set up 2 sockets
Aug 12 01:04:22 dist-test-slave-7f01 krb5kdc[28206](info): commencing operation
krb5kdc: starting...
W20260812 01:04:23.436966 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.371s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:04:23 dist-test-slave-7f01 krb5kdc[28206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:04:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:04:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 01:04:23.768285 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:42165
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:41495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:42165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:04:24.126965 28230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:04:24.127250 28230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:04:24.127417 28230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:04:24.134615 28230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:04:24.134719 28230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:04:24.134831 28230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:04:24.134958 28230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:04:24.135021 28230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:04:24.144959 28230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:41495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:42165
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:42165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:04:24.146273 28230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:04:24.147634 28230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:04:24.160089 28242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:24.160089 28237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:24.160151 28239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:04:24.160936 28230 server_base.cc:1034] running on GCE node
I20260812 01:04:24.161695 28230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:04:24.163084 28230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:04:24.164328 28230 hybrid_clock.cc:648] HybridClock initialized: now 1786496664164301 us; error 43 us; skew 500 ppm
I20260812 01:04:24.164682 28230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:04:24 dist-test-slave-7f01 krb5kdc[28206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496664, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:04:24.172300 28230 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 01:04:24.173821 28230 webserver.cc:460] Webserver started at http://127.21.201.126:46281/ using document root <none> and password file <none>
I20260812 01:04:24.174481 28230 fs_manager.cc:359] Metadata directory not provided
I20260812 01:04:24.174587 28230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:04:24.174907 28230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:04:24.177937 28230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "222a437b07624852bc5eab2b6fba88ee"
format_stamp: "Formatted at 2026-08-12 01:04:24 on dist-test-slave-7f01"
server_key: "0954478fe6e234848fe7a48dbfafdfa3"
server_key_iv: "6d5ffde94e423ebd657044d43e67a0f8"
server_key_version: "encryptionkey@0"
I20260812 01:04:24.178676 28230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "222a437b07624852bc5eab2b6fba88ee"
format_stamp: "Formatted at 2026-08-12 01:04:24 on dist-test-slave-7f01"
server_key: "0954478fe6e234848fe7a48dbfafdfa3"
server_key_iv: "6d5ffde94e423ebd657044d43e67a0f8"
server_key_version: "encryptionkey@0"
I20260812 01:04:24.183801 28230 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 01:04:24.187373 28253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:04:24.188745 28230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 01:04:24.188936 28230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "222a437b07624852bc5eab2b6fba88ee"
format_stamp: "Formatted at 2026-08-12 01:04:24 on dist-test-slave-7f01"
server_key: "0954478fe6e234848fe7a48dbfafdfa3"
server_key_iv: "6d5ffde94e423ebd657044d43e67a0f8"
server_key_version: "encryptionkey@0"
I20260812 01:04:24.189153 28230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:04:24.194851 28230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:04:24.199178 28230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:04:24.199502 28230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:04:24.218055 28230 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:42165
I20260812 01:04:24.218111 28341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:42165 every 8 connection(s)
I20260812 01:04:24.219480 28230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:04:24.221251 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 28230
I20260812 01:04:24.221511 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:04:24.222041 22309 external_mini_cluster.cc:1428] Setting key 237e6da5ccc81eaea5cd8ea79585f589
I20260812 01:04:24.223820 28342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:04:24.230994 28346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:24.224067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35277 (local address 127.21.201.126:42165)
0812 01:04:24.224932 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:24.224958 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:24.225013 (+ 55us) server_negotiation.cc:407] Connection header received
0812 01:04:24.227081 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:24.227109 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:24.227551 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:24.228028 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:24.229014 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:24.229025 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:24.229045 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:04:24.229064 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:24.229165 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:04:24.229195 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:24.229426 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:24.229549 (+ 123us) server_negotiation.cc:299] Negotiation successful
0812 01:04:24.229750 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":377,"threads_started":1}
I20260812 01:04:24.234781 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:04:24.238065 28342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 222a437b07624852bc5eab2b6fba88ee: Bootstrap starting.
I20260812 01:04:24.241953 28342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 222a437b07624852bc5eab2b6fba88ee: Neither blocks nor log segments found. Creating new log.
I20260812 01:04:24.241955 28346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:24.239408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41259 (local address 127.21.201.126:42165)
0812 01:04:24.239560 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:24.239565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:24.240006 (+ 441us) server_negotiation.cc:407] Connection header received
0812 01:04:24.240172 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:24.240176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:24.240229 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:24.240307 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:24.241274 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:24.241278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:24.241281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:04:24.241296 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:24.241333 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:04:24.241347 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:24.241433 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:24.241602 (+ 169us) server_negotiation.cc:299] Negotiation successful
0812 01:04:24.241649 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 01:04:24.243148 28342 log.cc:824] T 00000000000000000000000000000000 P 222a437b07624852bc5eab2b6fba88ee: Log is configured to *not* fsync() on all Append() calls
I20260812 01:04:24.245798 28342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 222a437b07624852bc5eab2b6fba88ee: No bootstrap required, opened a new log
I20260812 01:04:24.253033 28342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 222a437b07624852bc5eab2b6fba88ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222a437b07624852bc5eab2b6fba88ee" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42165 } }
I20260812 01:04:24.253435 28342 raft_consensus.cc:374] T 00000000000000000000000000000000 P 222a437b07624852bc5eab2b6fba88ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:04:24.253561 28342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 222a437b07624852bc5eab2b6fba88ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 222a437b07624852bc5eab2b6fba88ee, State: Initialized, Role: FOLLOWER
I20260812 01:04:24.254109 28342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 222a437b07624852bc5eab2b6fba88ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222a437b07624852bc5eab2b6fba88ee" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42165 } }
I20260812 01:04:24.254361 28342 raft_consensus.cc:388] T 00000000000000000000000000000000 P 222a437b07624852bc5eab2b6fba88ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:04:24.254520 28342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 222a437b07624852bc5eab2b6fba88ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:04:24.254696 28342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 222a437b07624852bc5eab2b6fba88ee [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:04:24.256958 28342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 222a437b07624852bc5eab2b6fba88ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222a437b07624852bc5eab2b6fba88ee" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42165 } }
I20260812 01:04:24.257462 28342 leader_election.cc:302] T 00000000000000000000000000000000 P 222a437b07624852bc5eab2b6fba88ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 222a437b07624852bc5eab2b6fba88ee; no voters:
I20260812 01:04:24.258062 28342 leader_election.cc:288] T 00000000000000000000000000000000 P 222a437b07624852bc5eab2b6fba88ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:04:24.258220 28358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 222a437b07624852bc5eab2b6fba88ee [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:04:24.258920 28358 raft_consensus.cc:686] T 00000000000000000000000000000000 P 222a437b07624852bc5eab2b6fba88ee [term 1 LEADER]: Becoming Leader. State: Replica: 222a437b07624852bc5eab2b6fba88ee, State: Running, Role: LEADER
I20260812 01:04:24.259500 28358 consensus_queue.cc:237] T 00000000000000000000000000000000 P 222a437b07624852bc5eab2b6fba88ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222a437b07624852bc5eab2b6fba88ee" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42165 } }
I20260812 01:04:24.260255 28342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 222a437b07624852bc5eab2b6fba88ee [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:04:24.262187 28359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 222a437b07624852bc5eab2b6fba88ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "222a437b07624852bc5eab2b6fba88ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222a437b07624852bc5eab2b6fba88ee" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42165 } } }
I20260812 01:04:24.262676 28359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 222a437b07624852bc5eab2b6fba88ee [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:24.262651 28361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 222a437b07624852bc5eab2b6fba88ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 222a437b07624852bc5eab2b6fba88ee. Latest consensus state: current_term: 1 leader_uuid: "222a437b07624852bc5eab2b6fba88ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222a437b07624852bc5eab2b6fba88ee" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42165 } } }
I20260812 01:04:24.263703 28361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 222a437b07624852bc5eab2b6fba88ee [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:24.264519 28370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:04:24.269785 28370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:04:24.280771 28370 catalog_manager.cc:1347] Generated new cluster ID: bd10e1c7c6c04601956291d54c99da3c
I20260812 01:04:24.280884 28370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:04:24.307894 28370 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:04:24.308441 28370 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:04:24.321710 28370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 222a437b07624852bc5eab2b6fba88ee: Generated new TSK 0
I20260812 01:04:24.322507 28370 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:04:24.335160 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 28230
W20260812 01:04:24.336309 28335 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35277
W20260812 01:04:24.337024 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 28230: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:04:24.337178 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 28230
2026-08-12T01:04:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:04:24 dist-test-slave-7f01 krb5kdc[28394](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:04:24 dist-test-slave-7f01 krb5kdc[28394](info): set up 2 sockets
Aug 12 01:04:24 dist-test-slave-7f01 krb5kdc[28394](info): commencing operation
krb5kdc: starting...
W20260812 01:04:25.781564 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.398s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:04:25 dist-test-slave-7f01 krb5kdc[28394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:04:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:04:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 01:04:26.109915 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:39307
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:33765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:39307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:04:26.462304 28413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:04:26.462598 28413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:04:26.462822 28413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:04:26.470049 28413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:04:26.470155 28413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:04:26.470203 28413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:04:26.470283 28413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:04:26.470320 28413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:04:26.480022 28413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:33765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:39307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:39307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:04:26.481266 28413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:04:26.482538 28413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:04:26.496145 28425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:26.496186 28426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:26.496342 28428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:04:26.497202 28413 server_base.cc:1034] running on GCE node
I20260812 01:04:26.498111 28413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:04:26.499543 28413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:04:26.500774 28413 hybrid_clock.cc:648] HybridClock initialized: now 1786496666500672 us; error 92 us; skew 500 ppm
I20260812 01:04:26.501128 28413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:04:26 dist-test-slave-7f01 krb5kdc[28394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496666, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:04:26.504565 28413 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 01:04:26.506057 28413 webserver.cc:460] Webserver started at http://127.21.201.126:46681/ using document root <none> and password file <none>
I20260812 01:04:26.506742 28413 fs_manager.cc:359] Metadata directory not provided
I20260812 01:04:26.506894 28413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:04:26.507223 28413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:04:26.510269 28413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "33fb51fc55af47819a61c22f58453f49"
format_stamp: "Formatted at 2026-08-12 01:04:26 on dist-test-slave-7f01"
server_key: "e91c89ef95686ac9face0e8b9befd048"
server_key_iv: "5c64dd999b85e54d6ca9bb691d06c2e5"
server_key_version: "encryptionkey@0"
I20260812 01:04:26.511051 28413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "33fb51fc55af47819a61c22f58453f49"
format_stamp: "Formatted at 2026-08-12 01:04:26 on dist-test-slave-7f01"
server_key: "e91c89ef95686ac9face0e8b9befd048"
server_key_iv: "5c64dd999b85e54d6ca9bb691d06c2e5"
server_key_version: "encryptionkey@0"
I20260812 01:04:26.516314 28413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 01:04:26.519904 28441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:04:26.521297 28413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:04:26.521497 28413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "33fb51fc55af47819a61c22f58453f49"
format_stamp: "Formatted at 2026-08-12 01:04:26 on dist-test-slave-7f01"
server_key: "e91c89ef95686ac9face0e8b9befd048"
server_key_iv: "5c64dd999b85e54d6ca9bb691d06c2e5"
server_key_version: "encryptionkey@0"
I20260812 01:04:26.521741 28413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:04:26.549258 28413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:04:26.553529 28413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:04:26.553838 28413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:04:26.573498 28413 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:39307
I20260812 01:04:26.573529 28532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:39307 every 8 connection(s)
I20260812 01:04:26.574884 28413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:04:26.578857 28534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:04:26.585088 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 28413
I20260812 01:04:26.585299 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:04:26.585762 22309 external_mini_cluster.cc:1428] Setting key c336a3c5bf4240e3d0e424a1b1c5fa62
I20260812 01:04:26.592401 28534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33fb51fc55af47819a61c22f58453f49: Bootstrap starting.
I20260812 01:04:26.596695 28534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33fb51fc55af47819a61c22f58453f49: Neither blocks nor log segments found. Creating new log.
I20260812 01:04:26.597898 28534 log.cc:824] T 00000000000000000000000000000000 P 33fb51fc55af47819a61c22f58453f49: Log is configured to *not* fsync() on all Append() calls
I20260812 01:04:26.600867 28534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33fb51fc55af47819a61c22f58453f49: No bootstrap required, opened a new log
Aug 12 01:04:26 dist-test-slave-7f01 krb5kdc[28394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:04:26.609004 28534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33fb51fc55af47819a61c22f58453f49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fb51fc55af47819a61c22f58453f49" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 39307 } }
I20260812 01:04:26.609411 28534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33fb51fc55af47819a61c22f58453f49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:04:26.609561 28534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33fb51fc55af47819a61c22f58453f49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33fb51fc55af47819a61c22f58453f49, State: Initialized, Role: FOLLOWER
I20260812 01:04:26.610224 28534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33fb51fc55af47819a61c22f58453f49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fb51fc55af47819a61c22f58453f49" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 39307 } }
I20260812 01:04:26.610566 28534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33fb51fc55af47819a61c22f58453f49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:04:26.610742 28534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33fb51fc55af47819a61c22f58453f49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:04:26.610939 28534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33fb51fc55af47819a61c22f58453f49 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:04:26.611868 28539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:26.587890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51169 (local address 127.21.201.126:39307)
0812 01:04:26.588884 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:26.588900 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:26.588958 (+ 58us) server_negotiation.cc:407] Connection header received
0812 01:04:26.591634 (+ 2676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:26.591666 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:26.592233 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:26.592709 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:26.594091 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:26.596411 (+ 2320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:26.597748 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:26.598887 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:26.599148 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:26.603898 (+ 4750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:26.603907 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:26.603921 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:26.603966 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:26.607349 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:26.608166 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:26.608178 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:26.608190 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:26.608381 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:26.608773 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:26.608782 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:26.608784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:26.609293 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:26.609645 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:26.610071 (+ 426us) server_negotiation.cc:299] Negotiation successful
0812 01:04:26.610529 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":423,"threads_started":1}
I20260812 01:04:26.613930 28534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33fb51fc55af47819a61c22f58453f49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fb51fc55af47819a61c22f58453f49" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 39307 } }
I20260812 01:04:26.614547 28534 leader_election.cc:302] T 00000000000000000000000000000000 P 33fb51fc55af47819a61c22f58453f49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33fb51fc55af47819a61c22f58453f49; no voters:
I20260812 01:04:26.615268 28534 leader_election.cc:288] T 00000000000000000000000000000000 P 33fb51fc55af47819a61c22f58453f49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:04:26.615376 28543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33fb51fc55af47819a61c22f58453f49 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:04:26.616077 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:04:26.617390 28534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33fb51fc55af47819a61c22f58453f49 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:04:26.619211 28543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33fb51fc55af47819a61c22f58453f49 [term 1 LEADER]: Becoming Leader. State: Replica: 33fb51fc55af47819a61c22f58453f49, State: Running, Role: LEADER
I20260812 01:04:26.620069 28543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33fb51fc55af47819a61c22f58453f49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fb51fc55af47819a61c22f58453f49" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 39307 } }
I20260812 01:04:26.624029 28544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33fb51fc55af47819a61c22f58453f49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33fb51fc55af47819a61c22f58453f49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fb51fc55af47819a61c22f58453f49" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 39307 } } }
I20260812 01:04:26.624408 28544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33fb51fc55af47819a61c22f58453f49 [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:26.624711 28545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33fb51fc55af47819a61c22f58453f49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33fb51fc55af47819a61c22f58453f49. Latest consensus state: current_term: 1 leader_uuid: "33fb51fc55af47819a61c22f58453f49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fb51fc55af47819a61c22f58453f49" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 39307 } } }
I20260812 01:04:26.626183 28545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33fb51fc55af47819a61c22f58453f49 [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:26.636612 28569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:04:26.642386 28569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:04:26.644655 28539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:26.624120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53527 (local address 127.21.201.126:39307)
0812 01:04:26.624319 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:26.624325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:26.624346 (+ 21us) server_negotiation.cc:407] Connection header received
0812 01:04:26.624511 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:26.624520 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:26.624634 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:26.624744 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:26.627788 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:26.629283 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:26.631809 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:26.632766 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:26.632923 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:26.637022 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:26.637030 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:26.637035 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:26.637076 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:26.641242 (+ 4166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:26.642140 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:26.642149 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:26.642152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:26.642318 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:26.642818 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:26.642823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:26.642826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:26.643307 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:26.643567 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:26.643989 (+ 422us) server_negotiation.cc:299] Negotiation successful
0812 01:04:26.644248 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 01:04:26.651014 28569 catalog_manager.cc:1347] Generated new cluster ID: d89bedea256a48a88a7b19f51102c94a
I20260812 01:04:26.651129 28569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:04:26.684306 28569 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:04:26.685772 28569 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:04:26.692798 28569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33fb51fc55af47819a61c22f58453f49: Generated new TSK 0
I20260812 01:04:26.693603 28569 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:04:26.700510 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 28413
I20260812 01:04:26.701709 28525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:04:26.796511 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 28413
2026-08-12T01:04:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2455 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:04:26 dist-test-slave-7f01 krb5kdc[28586](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:04:26 dist-test-slave-7f01 krb5kdc[28586](info): set up 2 sockets
Aug 12 01:04:26 dist-test-slave-7f01 krb5kdc[28586](info): commencing operation
krb5kdc: starting...
W20260812 01:04:28.172919 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.334s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:04:28 dist-test-slave-7f01 krb5kdc[28586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:04:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:04:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 01:04:28.495824 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:46005
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:34257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:46005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:04:28.863457 28609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:04:28.863845 28609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:04:28.864020 28609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:04:28.871472 28609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:04:28.871579 28609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:04:28.871628 28609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:04:28.871727 28609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:04:28.871840 28609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:04:28.881991 28609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:34257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:46005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:46005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:04:28.883373 28609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:04:28.884786 28609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:04:28.897285 28627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:28.897358 28624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:28.897333 28623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:04:28.898869 28609 server_base.cc:1034] running on GCE node
I20260812 01:04:28.899686 28609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:04:28.901211 28609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:04:28.902501 28609 hybrid_clock.cc:648] HybridClock initialized: now 1786496668902458 us; error 52 us; skew 500 ppm
I20260812 01:04:28.902912 28609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:04:28 dist-test-slave-7f01 krb5kdc[28586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496668, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:04:28.906175 28609 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 01:04:28.907848 28609 webserver.cc:460] Webserver started at http://127.21.201.126:33121/ using document root <none> and password file <none>
I20260812 01:04:28.908551 28609 fs_manager.cc:359] Metadata directory not provided
I20260812 01:04:28.908713 28609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:04:28.909044 28609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:04:28.912377 28609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "3d6cf4e6eb8640e8be2b437bef385e50"
format_stamp: "Formatted at 2026-08-12 01:04:28 on dist-test-slave-7f01"
server_key: "7fe21460eb96ccd38d80dc6998e2cf11"
server_key_iv: "2aef9dfff15dc31cd20b111cb5be8d65"
server_key_version: "encryptionkey@0"
I20260812 01:04:28.913225 28609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "3d6cf4e6eb8640e8be2b437bef385e50"
format_stamp: "Formatted at 2026-08-12 01:04:28 on dist-test-slave-7f01"
server_key: "7fe21460eb96ccd38d80dc6998e2cf11"
server_key_iv: "2aef9dfff15dc31cd20b111cb5be8d65"
server_key_version: "encryptionkey@0"
I20260812 01:04:28.918499 28609 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 01:04:28.922025 28638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:04:28.923336 28609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 01:04:28.923543 28609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "3d6cf4e6eb8640e8be2b437bef385e50"
format_stamp: "Formatted at 2026-08-12 01:04:28 on dist-test-slave-7f01"
server_key: "7fe21460eb96ccd38d80dc6998e2cf11"
server_key_iv: "2aef9dfff15dc31cd20b111cb5be8d65"
server_key_version: "encryptionkey@0"
I20260812 01:04:28.923794 28609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:04:28.939318 28609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:04:28.943460 28609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:04:28.943832 28609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:04:28.965720 28609 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:46005
I20260812 01:04:28.965755 28719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:46005 every 8 connection(s)
I20260812 01:04:28.967144 28609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:04:28.968047 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 28609
I20260812 01:04:28.968269 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:04:28.968786 22309 external_mini_cluster.cc:1428] Setting key 55c83e4ac1bce6f9a7aaf643b2c8e53b
I20260812 01:04:28.971508 28721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:04:28.982352 28724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:28.970640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59313 (local address 127.21.201.126:46005)
0812 01:04:28.971455 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:28.971467 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:28.971511 (+ 44us) server_negotiation.cc:407] Connection header received
0812 01:04:28.973264 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:28.973286 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:28.973631 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:28.973970 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:28.974593 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:28.976014 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:28.977842 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:28.978490 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:28.978624 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:28.979978 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:28.980007 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:28.980019 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:04:28.980548 (+ 529us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:28.980628 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:04:28.980655 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:28.980753 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:28.981079 (+ 326us) server_negotiation.cc:299] Negotiation successful
0812 01:04:28.981398 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":377,"threads_started":1}
I20260812 01:04:28.983982 28721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d6cf4e6eb8640e8be2b437bef385e50: Bootstrap starting.
I20260812 01:04:28.986251 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:04:28.987545 28721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d6cf4e6eb8640e8be2b437bef385e50: Neither blocks nor log segments found. Creating new log.
I20260812 01:04:28.988592 28721 log.cc:824] T 00000000000000000000000000000000 P 3d6cf4e6eb8640e8be2b437bef385e50: Log is configured to *not* fsync() on all Append() calls
I20260812 01:04:28.991899 28721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d6cf4e6eb8640e8be2b437bef385e50: No bootstrap required, opened a new log
I20260812 01:04:28.997063 28724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:28.990809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37899 (local address 127.21.201.126:46005)
0812 01:04:28.990994 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:28.991002 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:28.991444 (+ 442us) server_negotiation.cc:407] Connection header received
0812 01:04:28.991657 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:28.991661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:28.991718 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:28.991852 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:28.992495 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:28.993210 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:28.994523 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:28.994994 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:28.995153 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:28.996307 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:28.996312 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:28.996315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:04:28.996333 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:28.996372 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:04:28.996387 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:28.996502 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:28.996642 (+ 140us) server_negotiation.cc:299] Negotiation successful
0812 01:04:28.996737 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 01:04:29.000319 28721 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d6cf4e6eb8640e8be2b437bef385e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6cf4e6eb8640e8be2b437bef385e50" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46005 } }
I20260812 01:04:29.000685 28721 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d6cf4e6eb8640e8be2b437bef385e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:04:29.000803 28721 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d6cf4e6eb8640e8be2b437bef385e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d6cf4e6eb8640e8be2b437bef385e50, State: Initialized, Role: FOLLOWER
I20260812 01:04:29.001338 28721 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d6cf4e6eb8640e8be2b437bef385e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6cf4e6eb8640e8be2b437bef385e50" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46005 } }
I20260812 01:04:29.001581 28721 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d6cf4e6eb8640e8be2b437bef385e50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:04:29.001736 28721 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d6cf4e6eb8640e8be2b437bef385e50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:04:29.001914 28721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d6cf4e6eb8640e8be2b437bef385e50 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:04:29.004330 28721 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d6cf4e6eb8640e8be2b437bef385e50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6cf4e6eb8640e8be2b437bef385e50" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46005 } }
I20260812 01:04:29.004832 28721 leader_election.cc:302] T 00000000000000000000000000000000 P 3d6cf4e6eb8640e8be2b437bef385e50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d6cf4e6eb8640e8be2b437bef385e50; no voters:
I20260812 01:04:29.005429 28721 leader_election.cc:288] T 00000000000000000000000000000000 P 3d6cf4e6eb8640e8be2b437bef385e50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:04:29.005587 28734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d6cf4e6eb8640e8be2b437bef385e50 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:04:29.006362 28734 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d6cf4e6eb8640e8be2b437bef385e50 [term 1 LEADER]: Becoming Leader. State: Replica: 3d6cf4e6eb8640e8be2b437bef385e50, State: Running, Role: LEADER
I20260812 01:04:29.007026 28734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d6cf4e6eb8640e8be2b437bef385e50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6cf4e6eb8640e8be2b437bef385e50" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46005 } }
I20260812 01:04:29.007561 28721 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d6cf4e6eb8640e8be2b437bef385e50 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:04:29.010041 28737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d6cf4e6eb8640e8be2b437bef385e50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d6cf4e6eb8640e8be2b437bef385e50. Latest consensus state: current_term: 1 leader_uuid: "3d6cf4e6eb8640e8be2b437bef385e50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6cf4e6eb8640e8be2b437bef385e50" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46005 } } }
I20260812 01:04:29.011022 28737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d6cf4e6eb8640e8be2b437bef385e50 [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:29.010819 28735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d6cf4e6eb8640e8be2b437bef385e50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d6cf4e6eb8640e8be2b437bef385e50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6cf4e6eb8640e8be2b437bef385e50" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 46005 } } }
I20260812 01:04:29.011317 28735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d6cf4e6eb8640e8be2b437bef385e50 [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:29.018797 28747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:04:29.023295 28747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:04:29.031975 28747 catalog_manager.cc:1347] Generated new cluster ID: 3d6ef8f08f4c42bebacbe776e323cb2a
I20260812 01:04:29.032087 28747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:04:29.044549 28747 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:04:29.045862 28747 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:04:29.057962 28747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d6cf4e6eb8640e8be2b437bef385e50: Generated new TSK 0
I20260812 01:04:29.058993 28747 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:04:29.077070 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 28609
W20260812 01:04:29.078164 28711 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59313
W20260812 01:04:29.078903 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 28609: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:04:29.079056 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 28609
2026-08-12T01:04:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:04:29 dist-test-slave-7f01 krb5kdc[28770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:04:29 dist-test-slave-7f01 krb5kdc[28770](info): set up 2 sockets
Aug 12 01:04:29 dist-test-slave-7f01 krb5kdc[28770](info): commencing operation
krb5kdc: starting...
W20260812 01:04:30.478047 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.351s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:04:30 dist-test-slave-7f01 krb5kdc[28770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:04:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:04:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 01:04:30.811928 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:39151
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:39151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:04:31.164556 28794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:04:31.164911 28794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:04:31.165088 28794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:04:31.172616 28794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:04:31.172775 28794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:04:31.172883 28794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:04:31.173002 28794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:04:31.173082 28794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:04:31.183038 28794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:39151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:39151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:04:31.184437 28794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:04:31.185783 28794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:04:31.197938 28802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:31.197986 28804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:31.197880 28807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:04:31.199011 28794 server_base.cc:1034] running on GCE node
I20260812 01:04:31.199808 28794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:04:31.201247 28794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:04:31.202462 28794 hybrid_clock.cc:648] HybridClock initialized: now 1786496671202407 us; error 50 us; skew 500 ppm
I20260812 01:04:31.202821 28794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:04:31 dist-test-slave-7f01 krb5kdc[28770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496671, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:04:31.206257 28794 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 01:04:31.207723 28794 webserver.cc:460] Webserver started at http://127.21.201.126:37943/ using document root <none> and password file <none>
I20260812 01:04:31.208452 28794 fs_manager.cc:359] Metadata directory not provided
I20260812 01:04:31.208578 28794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:04:31.208832 28794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:04:31.211948 28794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "d678f6b9166b4e029e471befe95aed6a"
format_stamp: "Formatted at 2026-08-12 01:04:31 on dist-test-slave-7f01"
server_key: "076d36a247cb1a7a60e2b2f69e7ae5e4"
server_key_iv: "1998fec060895f47064291fe6ab3a175"
server_key_version: "encryptionkey@0"
I20260812 01:04:31.212688 28794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "d678f6b9166b4e029e471befe95aed6a"
format_stamp: "Formatted at 2026-08-12 01:04:31 on dist-test-slave-7f01"
server_key: "076d36a247cb1a7a60e2b2f69e7ae5e4"
server_key_iv: "1998fec060895f47064291fe6ab3a175"
server_key_version: "encryptionkey@0"
I20260812 01:04:31.217959 28794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260812 01:04:31.221418 28819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:04:31.222687 28794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 01:04:31.222888 28794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "d678f6b9166b4e029e471befe95aed6a"
format_stamp: "Formatted at 2026-08-12 01:04:31 on dist-test-slave-7f01"
server_key: "076d36a247cb1a7a60e2b2f69e7ae5e4"
server_key_iv: "1998fec060895f47064291fe6ab3a175"
server_key_version: "encryptionkey@0"
I20260812 01:04:31.223093 28794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:04:31.228863 28794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:04:31.233233 28794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:04:31.233537 28794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:04:31.252262 28794 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:39151
I20260812 01:04:31.252311 28897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:39151 every 8 connection(s)
I20260812 01:04:31.253598 28794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:04:31.254796 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 28794
I20260812 01:04:31.254997 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:04:31.255553 22309 external_mini_cluster.cc:1428] Setting key 2d471c886de130504ac898dcb450cfce
I20260812 01:04:31.258029 28898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:04:31.263298 28903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:31.257320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45007 (local address 127.21.201.126:39151)
0812 01:04:31.258177 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:31.258189 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:31.258234 (+ 45us) server_negotiation.cc:407] Connection header received
0812 01:04:31.259903 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:31.259924 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:31.260231 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:31.260557 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:31.261659 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:31.261666 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:31.261679 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:04:31.261697 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:31.261794 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:04:31.261818 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:31.262025 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:31.262151 (+ 126us) server_negotiation.cc:299] Negotiation successful
0812 01:04:31.262373 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":353,"threads_started":1}
I20260812 01:04:31.267189 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:04:31.271471 28898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d678f6b9166b4e029e471befe95aed6a: Bootstrap starting.
I20260812 01:04:31.275146 28898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d678f6b9166b4e029e471befe95aed6a: Neither blocks nor log segments found. Creating new log.
I20260812 01:04:31.275152 28903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:31.272445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58217 (local address 127.21.201.126:39151)
0812 01:04:31.272622 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:31.272627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:31.273158 (+ 531us) server_negotiation.cc:407] Connection header received
0812 01:04:31.273255 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:31.273260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:31.273331 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:31.273472 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:31.274364 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:31.274369 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:31.274374 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 01:04:31.274391 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:31.274451 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 01:04:31.274472 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:31.274585 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:31.274763 (+ 178us) server_negotiation.cc:299] Negotiation successful
0812 01:04:31.274819 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 01:04:31.276530 28898 log.cc:824] T 00000000000000000000000000000000 P d678f6b9166b4e029e471befe95aed6a: Log is configured to *not* fsync() on all Append() calls
I20260812 01:04:31.279186 28898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d678f6b9166b4e029e471befe95aed6a: No bootstrap required, opened a new log
I20260812 01:04:31.286891 28898 raft_consensus.cc:348] T 00000000000000000000000000000000 P d678f6b9166b4e029e471befe95aed6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d678f6b9166b4e029e471befe95aed6a" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 39151 } }
I20260812 01:04:31.287290 28898 raft_consensus.cc:374] T 00000000000000000000000000000000 P d678f6b9166b4e029e471befe95aed6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 01:04:31.287407 28898 raft_consensus.cc:729] T 00000000000000000000000000000000 P d678f6b9166b4e029e471befe95aed6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d678f6b9166b4e029e471befe95aed6a, State: Initialized, Role: FOLLOWER
I20260812 01:04:31.288012 28898 consensus_queue.cc:260] T 00000000000000000000000000000000 P d678f6b9166b4e029e471befe95aed6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d678f6b9166b4e029e471befe95aed6a" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 39151 } }
I20260812 01:04:31.288264 28898 raft_consensus.cc:388] T 00000000000000000000000000000000 P d678f6b9166b4e029e471befe95aed6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 01:04:31.288419 28898 raft_consensus.cc:482] T 00000000000000000000000000000000 P d678f6b9166b4e029e471befe95aed6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 01:04:31.288596 28898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d678f6b9166b4e029e471befe95aed6a [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:04:31.290707 28898 raft_consensus.cc:504] T 00000000000000000000000000000000 P d678f6b9166b4e029e471befe95aed6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d678f6b9166b4e029e471befe95aed6a" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 39151 } }
I20260812 01:04:31.291198 28898 leader_election.cc:302] T 00000000000000000000000000000000 P d678f6b9166b4e029e471befe95aed6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d678f6b9166b4e029e471befe95aed6a; no voters:
I20260812 01:04:31.291955 28898 leader_election.cc:288] T 00000000000000000000000000000000 P d678f6b9166b4e029e471befe95aed6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 01:04:31.292057 28911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d678f6b9166b4e029e471befe95aed6a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:04:31.292721 28911 raft_consensus.cc:686] T 00000000000000000000000000000000 P d678f6b9166b4e029e471befe95aed6a [term 1 LEADER]: Becoming Leader. State: Replica: d678f6b9166b4e029e471befe95aed6a, State: Running, Role: LEADER
I20260812 01:04:31.293292 28911 consensus_queue.cc:237] T 00000000000000000000000000000000 P d678f6b9166b4e029e471befe95aed6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d678f6b9166b4e029e471befe95aed6a" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 39151 } }
I20260812 01:04:31.294153 28898 sys_catalog.cc:564] T 00000000000000000000000000000000 P d678f6b9166b4e029e471befe95aed6a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:04:31.296269 28914 sys_catalog.cc:455] T 00000000000000000000000000000000 P d678f6b9166b4e029e471befe95aed6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d678f6b9166b4e029e471befe95aed6a. Latest consensus state: current_term: 1 leader_uuid: "d678f6b9166b4e029e471befe95aed6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d678f6b9166b4e029e471befe95aed6a" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 39151 } } }
I20260812 01:04:31.296681 28914 sys_catalog.cc:458] T 00000000000000000000000000000000 P d678f6b9166b4e029e471befe95aed6a [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:31.296429 28912 sys_catalog.cc:455] T 00000000000000000000000000000000 P d678f6b9166b4e029e471befe95aed6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d678f6b9166b4e029e471befe95aed6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d678f6b9166b4e029e471befe95aed6a" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 39151 } } }
I20260812 01:04:31.297240 28912 sys_catalog.cc:458] T 00000000000000000000000000000000 P d678f6b9166b4e029e471befe95aed6a [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:31.299513 28925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:04:31.304908 28925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:04:31.312861 28925 catalog_manager.cc:1347] Generated new cluster ID: 2c664bd9e7b64e2a8f2bc42b58ba7e0e
I20260812 01:04:31.312978 28925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:04:31.335300 28925 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:04:31.335712 28925 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:04:31.343652 28925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d678f6b9166b4e029e471befe95aed6a: Generated new TSK 0
I20260812 01:04:31.344369 28925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:04:31.356138 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 28794
W20260812 01:04:31.357179 28893 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45007
W20260812 01:04:31.357863 22309 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 28794: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 01:04:31.358033 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 28794
2026-08-12T01:04:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2269 ms)
[----------] 18 tests from AuthTokenIssuingTest (42145 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:04:31 dist-test-slave-7f01 krb5kdc[28953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:04:31 dist-test-slave-7f01 krb5kdc[28953](info): set up 2 sockets
Aug 12 01:04:31 dist-test-slave-7f01 krb5kdc[28953](info): commencing operation
krb5kdc: starting...
W20260812 01:04:32.742319 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.345s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:04:32 dist-test-slave-7f01 krb5kdc[28953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:04:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:04:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 01:04:33.068003 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:41499
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:41137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:41499,127.21.201.125:44351,127.21.201.124:35273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:04:33.407341 28974 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 01:04:33.407703 28974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:04:33.407886 28974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:04:33.408036 28974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:04:33.415215 28974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:04:33.415294 28974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:04:33.415398 28974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:04:33.415499 28974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:04:33.415596 28974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:04:33.425241 28974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:41137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:41499,127.21.201.125:44351,127.21.201.124:35273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:41499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:04:33.426503 28974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:04:33.427867 28974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:04:33.440052 28985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:33.440141 28981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:33.440179 28982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:04:33.440894 28974 server_base.cc:1034] running on GCE node
I20260812 01:04:33.441670 28974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:04:33.442948 28974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:04:33.444201 28974 hybrid_clock.cc:648] HybridClock initialized: now 1786496673444145 us; error 33 us; skew 500 ppm
I20260812 01:04:33.444551 28974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:04:33 dist-test-slave-7f01 krb5kdc[28953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496673, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:04:33.447886 28974 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 01:04:33.449359 28974 webserver.cc:460] Webserver started at http://127.21.201.126:37409/ using document root <none> and password file <none>
I20260812 01:04:33.450057 28974 fs_manager.cc:359] Metadata directory not provided
I20260812 01:04:33.450222 28974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:04:33.450587 28974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:04:33.454350 28974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "875f42f7d1c74e89beae9721c18e389e"
format_stamp: "Formatted at 2026-08-12 01:04:33 on dist-test-slave-7f01"
server_key: "59f35e0fe7ae78a1d2180afc40b6644e"
server_key_iv: "af3347cca170b48871362a3ffe38aa22"
server_key_version: "encryptionkey@0"
I20260812 01:04:33.455551 28974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "875f42f7d1c74e89beae9721c18e389e"
format_stamp: "Formatted at 2026-08-12 01:04:33 on dist-test-slave-7f01"
server_key: "59f35e0fe7ae78a1d2180afc40b6644e"
server_key_iv: "af3347cca170b48871362a3ffe38aa22"
server_key_version: "encryptionkey@0"
I20260812 01:04:33.460865 28974 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260812 01:04:33.464352 28995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:04:33.465583 28974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 01:04:33.465768 28974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "875f42f7d1c74e89beae9721c18e389e"
format_stamp: "Formatted at 2026-08-12 01:04:33 on dist-test-slave-7f01"
server_key: "59f35e0fe7ae78a1d2180afc40b6644e"
server_key_iv: "af3347cca170b48871362a3ffe38aa22"
server_key_version: "encryptionkey@0"
I20260812 01:04:33.465997 28974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:04:33.490857 28974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:04:33.495126 28974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:04:33.495445 28974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:04:33.515331 28974 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:41499
I20260812 01:04:33.515352 29093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:41499 every 8 connection(s)
I20260812 01:04:33.516814 28974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:04:33.518828 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 28974
I20260812 01:04:33.519075 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:04:33.519980 22309 external_mini_cluster.cc:1428] Setting key 73d97425cd84528bf83220d66a9c4e64
I20260812 01:04:33.521610 29094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:04:33.531903 29094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41499 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:04:33 dist-test-slave-7f01 krb5kdc[28953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496673, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.201.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.125@KRBTEST.COM" created.
I20260812 01:04:33.555423 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:33.535926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:41499 (local address 127.0.0.1:39764)
0812 01:04:33.536756 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:33.536801 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 01:04:33.538993 (+ 2192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:04:33.539833 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:04:33.539840 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:04:33.540577 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:04:33.540919 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:33.540937 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:33.542513 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:33.542523 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:33.543568 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:33.543581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:33.544537 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:33.544544 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:33.544759 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:33.545459 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:04:33.545487 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:04:33.547829 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:04:33.551310 (+ 3481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:33.551324 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:33.551337 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:33.551793 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:33.552214 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:33.552220 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:33.552223 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:33.552393 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:33.553327 (+ 934us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:04:33.553334 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:04:33.553614 (+ 280us) client_negotiation.cc:769] Sending connection context
0812 01:04:33.553890 (+ 276us) client_negotiation.cc:240] Negotiation successful
0812 01:04:33.554310 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":326,"threads_started":1}
I20260812 01:04:33.555508 29098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:33.537162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39764 (local address 127.21.201.126:41499)
0812 01:04:33.538164 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:33.538172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:33.538225 (+ 53us) server_negotiation.cc:407] Connection header received
0812 01:04:33.539360 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:33.539368 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:33.539575 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:33.539828 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:33.541140 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:33.542345 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:33.543715 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:33.544300 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:33.544505 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:33.548050 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:33.548058 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:33.548069 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:33.548102 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:33.551157 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:33.551939 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:33.551945 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:33.551948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:33.552099 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:33.552570 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:33.552576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:33.552580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:33.553159 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:33.553326 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:33.554073 (+ 747us) server_negotiation.cc:299] Negotiation successful
0812 01:04:33.554479 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":795,"thread_start_us":602,"threads_started":1}
I20260812 01:04:33.558796 29094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 } has no permanent_uuid. Determining permanent_uuid...
I20260812 01:04:33.560089 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:33.559648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:44351 (local address 127.0.0.1:53522)
0812 01:04:33.559853 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:33.559978 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260812 01:04:33.560451 29001 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.201.125:44351: connect: Connection refused (error 111)
I20260812 01:04:33.562000 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:33.561663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:44351 (local address 127.0.0.1:53528)
0812 01:04:33.561798 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:33.561864 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 01:04:33.562295 29094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.125:44351: Network error: Client connection negotiation failed: client connection to 127.21.201.125:44351: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.125.keytab.
Entry for principal kudu/127.21.201.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.125.keytab.
I20260812 01:04:33.594935 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.125:44351
--webserver_interface=127.21.201.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.125.keytab
--principal=kudu/127.21.201.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:41137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:41499,127.21.201.125:44351,127.21.201.124:35273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:04:33.611665 29094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 } attempt: 1
I20260812 01:04:33.613142 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:33.612540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:44351 (local address 127.0.0.1:53538)
0812 01:04:33.612877 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:33.613004 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260812 01:04:33.614259 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:33.613917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:44351 (local address 127.0.0.1:53554)
0812 01:04:33.614053 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:33.614127 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 01:04:33.614549 29094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.125:44351: Network error: Client connection negotiation failed: client connection to 127.21.201.125:44351: connect: Connection refused (error 111)
I20260812 01:04:33.682814 29094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 } attempt: 2
I20260812 01:04:33.684141 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:33.683579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:44351 (local address 127.0.0.1:53566)
0812 01:04:33.683892 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:33.683990 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260812 01:04:33.685184 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:33.684811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:44351 (local address 127.0.0.1:53574)
0812 01:04:33.684961 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:33.685050 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 01:04:33.685382 29094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.125:44351: Network error: Client connection negotiation failed: client connection to 127.21.201.125:44351: connect: Connection refused (error 111)
I20260812 01:04:33.776646 29094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 } attempt: 3
I20260812 01:04:33.777825 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:33.777370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:44351 (local address 127.0.0.1:53584)
0812 01:04:33.777587 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:33.777688 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260812 01:04:33.778867 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:33.778509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:44351 (local address 127.0.0.1:53600)
0812 01:04:33.778672 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:33.778736 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260812 01:04:33.779063 29094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.125:44351: Network error: Client connection negotiation failed: client connection to 127.21.201.125:44351: connect: Connection refused (error 111)
I20260812 01:04:33.922338 29094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 } attempt: 4
I20260812 01:04:33.923671 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:33.923075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:44351 (local address 127.0.0.1:53602)
0812 01:04:33.923434 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:33.923527 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260812 01:04:33.924799 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:33.924429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:44351 (local address 127.0.0.1:53612)
0812 01:04:33.924578 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:33.924672 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 01:04:33.924996 29094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.125:44351: Network error: Client connection negotiation failed: client connection to 127.21.201.125:44351: connect: Connection refused (error 111)
W20260812 01:04:33.946975 29104 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 01:04:33.947343 29104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:04:33.947479 29104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:04:33.947623 29104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:04:33.954773 29104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:04:33.954911 29104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:04:33.955047 29104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:04:33.955148 29104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:04:33.955242 29104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:04:33.965010 29104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:41137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:41499,127.21.201.125:44351,127.21.201.124:35273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.125.keytab
--principal=kudu/127.21.201.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.125:44351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.201.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:04:33.966437 29104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:04:33.967773 29104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:04:33.979835 29112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:33.980062 29117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:33.980134 29113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:04:33.985219 29104 server_base.cc:1034] running on GCE node
I20260812 01:04:33.986037 29104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:04:33.987490 29104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:04:33.988718 29104 hybrid_clock.cc:648] HybridClock initialized: now 1786496673988624 us; error 63 us; skew 500 ppm
I20260812 01:04:33.989022 29104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:04:33 dist-test-slave-7f01 krb5kdc[28953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496673, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:04:33.992437 29104 init.cc:377] Logged in from keytab as kudu/127.21.201.125@KRBTEST.COM (short username kudu)
I20260812 01:04:33.993868 29104 webserver.cc:460] Webserver started at http://127.21.201.125:36011/ using document root <none> and password file <none>
I20260812 01:04:33.994539 29104 fs_manager.cc:359] Metadata directory not provided
I20260812 01:04:33.994632 29104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:04:33.994910 29104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:04:33.998210 29104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-1/data/instance:
uuid: "ee5626ae096042e187f7e7e860b0228f"
format_stamp: "Formatted at 2026-08-12 01:04:33 on dist-test-slave-7f01"
server_key: "66335d0b4f0786f1efb0d0dee20555e9"
server_key_iv: "f01e5ab71966fa5e1f9591f2e8fb75a0"
server_key_version: "encryptionkey@0"
I20260812 01:04:33.999019 29104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-1/wal/instance:
uuid: "ee5626ae096042e187f7e7e860b0228f"
format_stamp: "Formatted at 2026-08-12 01:04:33 on dist-test-slave-7f01"
server_key: "66335d0b4f0786f1efb0d0dee20555e9"
server_key_iv: "f01e5ab71966fa5e1f9591f2e8fb75a0"
server_key_version: "encryptionkey@0"
I20260812 01:04:34.004201 29104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260812 01:04:34.007552 29126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:04:34.008888 29104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:04:34.009078 29104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-1/wal
uuid: "ee5626ae096042e187f7e7e860b0228f"
format_stamp: "Formatted at 2026-08-12 01:04:33 on dist-test-slave-7f01"
server_key: "66335d0b4f0786f1efb0d0dee20555e9"
server_key_iv: "f01e5ab71966fa5e1f9591f2e8fb75a0"
server_key_version: "encryptionkey@0"
I20260812 01:04:34.009325 29104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:04:34.031427 29104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:04:34.036154 29104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:04:34.036465 29104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:04:34.056118 29104 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.125:44351
I20260812 01:04:34.056142 29202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.125:44351 every 8 connection(s)
I20260812 01:04:34.057495 29104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-1/data/info.pb
I20260812 01:04:34.057926 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 29104
I20260812 01:04:34.058115 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-1/wal/instance
I20260812 01:04:34.058710 22309 external_mini_cluster.cc:1428] Setting key 4c197721652dacdbc59afaf4c82f7fc3
I20260812 01:04:34.062068 29204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:04:34.072048 29204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41499 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:04:34 dist-test-slave-7f01 krb5kdc[28953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496673, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.125@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.201.124@KRBTEST.COM; defaulting to no policy
I20260812 01:04:34.093168 29209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.075190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39776 (local address 127.21.201.126:41499)
0812 01:04:34.075701 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:34.075707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:34.076702 (+ 995us) server_negotiation.cc:407] Connection header received
0812 01:04:34.078602 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:34.078608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:34.078701 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:34.078827 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:34.080407 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.081130 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.082646 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.083066 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.083222 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:34.087049 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:34.087056 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:34.087061 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:34.087119 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:34.090019 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.090765 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.090773 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.090781 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.090904 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.091339 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.091348 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.091350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.091618 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:34.091818 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:34.092480 (+ 662us) server_negotiation.cc:299] Negotiation successful
0812 01:04:34.092706 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":281,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.124@KRBTEST.COM" created.
I20260812 01:04:34.094183 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.075812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:41499 (local address 127.0.0.1:39776)
0812 01:04:34.076542 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:34.076583 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 01:04:34.078320 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:04:34.079069 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:04:34.079076 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:04:34.079584 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:04:34.080186 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:34.080203 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.081309 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.081313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:34.082495 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:34.082506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.083314 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.083322 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:34.083563 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:34.084337 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:04:34.084358 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:04:34.086790 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:04:34.090171 (+ 3381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:34.090179 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:34.090193 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:34.090630 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:34.091029 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:34.091034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:34.091038 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:34.091209 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:34.091774 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:04:34.091782 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:04:34.092125 (+ 343us) client_negotiation.cc:769] Sending connection context
0812 01:04:34.092383 (+ 258us) client_negotiation.cc:240] Negotiation successful
0812 01:04:34.092856 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":305,"threads_started":1}
I20260812 01:04:34.096028 29204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:04:34 dist-test-slave-7f01 krb5kdc[28953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496673, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.125@KRBTEST.COM for kudu/127.21.201.125@KRBTEST.COM
I20260812 01:04:34.111176 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.096869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:44351 (local address 127.0.0.1:53620)
0812 01:04:34.097042 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:34.097065 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 01:04:34.097162 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:04:34.098241 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:04:34.098245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:04:34.098690 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:04:34.098815 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:34.098823 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.100214 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.100222 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:34.101049 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:34.101060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.101856 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.101861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:34.102070 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:34.102653 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:04:34.102669 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:04:34.104747 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:04:34.108579 (+ 3832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:34.108583 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:34.108587 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:34.108955 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:34.109356 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:34.109362 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:34.109365 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:34.109513 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:34.110171 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:04:34.110180 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:04:34.110340 (+ 160us) client_negotiation.cc:769] Sending connection context
0812 01:04:34.110487 (+ 147us) client_negotiation.cc:240] Negotiation successful
0812 01:04:34.110773 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 01:04:34.111270 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.097158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53620 (local address 127.21.201.125:44351)
0812 01:04:34.097863 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:34.097869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:34.097892 (+ 23us) server_negotiation.cc:407] Connection header received
0812 01:04:34.097932 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:34.097938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:34.098094 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:34.098244 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:34.098998 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.100085 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.101192 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.101713 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.101849 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:34.104955 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:34.104966 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:34.104977 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:34.105012 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:34.108393 (+ 3381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.109102 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.109110 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.109115 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.109242 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.109624 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.109630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.109633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.110043 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:34.110271 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:34.110616 (+ 345us) server_negotiation.cc:299] Negotiation successful
0812 01:04:34.110882 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":559,"thread_start_us":419,"threads_started":1}
I20260812 01:04:34.113786 29204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } has no permanent_uuid. Determining permanent_uuid...
I20260812 01:04:34.114995 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.114562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:35273 (local address 127.0.0.1:34192)
0812 01:04:34.114744 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:34.114838 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260812 01:04:34.115278 29129 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
I20260812 01:04:34.116907 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.116570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:35273 (local address 127.0.0.1:34200)
0812 01:04:34.116744 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:34.116813 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 01:04:34.117151 29204 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:35273: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.124.keytab.
Entry for principal kudu/127.21.201.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.124.keytab.
I20260812 01:04:34.132234 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.124:35273
--webserver_interface=127.21.201.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.124.keytab
--principal=kudu/127.21.201.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:41137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:41499,127.21.201.125:44351,127.21.201.124:35273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:04:34.166538 29204 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } attempt: 1
I20260812 01:04:34.167941 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.167342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:35273 (local address 127.0.0.1:34206)
0812 01:04:34.167543 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:34.167678 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260812 01:04:34.168978 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.168619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:35273 (local address 127.0.0.1:34216)
0812 01:04:34.168800 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:34.168877 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 01:04:34.169268 29204 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:35273: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
I20260812 01:04:34.224272 29094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 } attempt: 5
Aug 12 01:04:34 dist-test-slave-7f01 krb5kdc[28953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496673, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for kudu/127.21.201.125@KRBTEST.COM
I20260812 01:04:34.237516 29204 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } attempt: 2
I20260812 01:04:34.238601 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.238063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:35273 (local address 127.0.0.1:34224)
0812 01:04:34.238247 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:34.238355 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 01:04:34.239779 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.239351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:35273 (local address 127.0.0.1:34230)
0812 01:04:34.239512 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:34.239600 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 01:04:34.240051 29204 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:35273: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
I20260812 01:04:34.244200 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.225069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:44351 (local address 127.0.0.1:53624)
0812 01:04:34.225274 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:34.225298 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 01:04:34.225420 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:04:34.225823 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:04:34.225827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:04:34.226229 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:04:34.226415 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:34.226424 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.228033 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.228038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:34.228910 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:34.228921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.229943 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.229947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:34.230113 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:34.234959 (+ 4846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:04:34.234984 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:04:34.237249 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:04:34.241864 (+ 4615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:34.241869 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:34.241871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:34.242248 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:34.242672 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:34.242675 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:34.242677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:34.242812 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:34.243415 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:04:34.243418 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:04:34.243542 (+ 124us) client_negotiation.cc:769] Sending connection context
0812 01:04:34.243660 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 01:04:34.243968 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260812 01:04:34.244335 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.225167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53624 (local address 127.21.201.125:44351)
0812 01:04:34.225378 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:34.225385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:34.225429 (+ 44us) server_negotiation.cc:407] Connection header received
0812 01:04:34.225574 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:34.225579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:34.225662 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:34.225789 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:34.226575 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.227890 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.229073 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.229799 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.229934 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:34.237511 (+ 7577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:34.237519 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:34.237524 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:34.237567 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:34.241712 (+ 4145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.242379 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.242385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.242390 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.242559 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.242924 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.242929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.242932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.243292 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:34.243450 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:34.243832 (+ 382us) server_negotiation.cc:299] Negotiation successful
0812 01:04:34.244053 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260812 01:04:34.245258 29094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } has no permanent_uuid. Determining permanent_uuid...
I20260812 01:04:34.246332 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.245963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:35273 (local address 127.0.0.1:34234)
0812 01:04:34.246130 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:34.246213 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 01:04:34.247370 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.247037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:35273 (local address 127.0.0.1:34244)
0812 01:04:34.247181 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:34.247265 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 01:04:34.247607 29094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:35273: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
I20260812 01:04:34.298842 29094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } attempt: 1
I20260812 01:04:34.300367 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.299674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:35273 (local address 127.0.0.1:34256)
0812 01:04:34.299955 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:34.300102 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260812 01:04:34.301580 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.301171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:35273 (local address 127.0.0.1:34258)
0812 01:04:34.301339 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:34.301441 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260812 01:04:34.301820 29094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:35273: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
I20260812 01:04:34.331414 29204 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } attempt: 3
I20260812 01:04:34.332723 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.332220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:35273 (local address 127.0.0.1:34260)
0812 01:04:34.332426 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:34.332539 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260812 01:04:34.333824 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.333410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:35273 (local address 127.0.0.1:34266)
0812 01:04:34.333637 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:34.333721 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260812 01:04:34.334095 29204 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:35273: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
I20260812 01:04:34.370057 29094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } attempt: 2
I20260812 01:04:34.371344 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.370821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:35273 (local address 127.0.0.1:34282)
0812 01:04:34.371062 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:34.371163 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260812 01:04:34.372471 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.372126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:35273 (local address 127.0.0.1:34296)
0812 01:04:34.372273 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:34.372365 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 01:04:34.372704 29094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:35273: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
I20260812 01:04:34.477362 29204 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } attempt: 4
I20260812 01:04:34.478708 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.478115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:35273 (local address 127.0.0.1:34298)
0812 01:04:34.478359 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:34.478480 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260812 01:04:34.478940 29094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } attempt: 3
I20260812 01:04:34.480064 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.479527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:35273 (local address 127.0.0.1:34306)
0812 01:04:34.479717 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:34.479875 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260812 01:04:34.480159 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.479792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:35273 (local address 127.0.0.1:34316)
0812 01:04:34.479934 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:34.480012 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 01:04:34.480603 29204 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:35273: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
I20260812 01:04:34.481160 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.480764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:35273 (local address 127.0.0.1:34330)
0812 01:04:34.480981 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:34.481059 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260812 01:04:34.481387 29094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:35273: Network error: Client connection negotiation failed: client connection to 127.21.201.124:35273: connect: Connection refused (error 111)
W20260812 01:04:34.487278 29216 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 01:04:34.487622 29216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:04:34.487733 29216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:04:34.487908 29216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:04:34.495097 29216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:04:34.495177 29216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:04:34.495234 29216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:04:34.495285 29216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:04:34.495350 29216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:04:34.505187 29216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:41137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:41499,127.21.201.125:44351,127.21.201.124:35273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.124.keytab
--principal=kudu/127.21.201.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.124:35273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.201.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:04:34.506377 29216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:04:34.507661 29216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:04:34.519486 29226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:34.519778 29228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:34.519452 29224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:04:34.520829 29216 server_base.cc:1034] running on GCE node
I20260812 01:04:34.521615 29216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:04:34.522989 29216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:04:34.524230 29216 hybrid_clock.cc:648] HybridClock initialized: now 1786496674524145 us; error 54 us; skew 500 ppm
I20260812 01:04:34.524577 29216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:04:34 dist-test-slave-7f01 krb5kdc[28953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496674, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:04:34.527786 29216 init.cc:377] Logged in from keytab as kudu/127.21.201.124@KRBTEST.COM (short username kudu)
I20260812 01:04:34.529246 29216 webserver.cc:460] Webserver started at http://127.21.201.124:37835/ using document root <none> and password file <none>
I20260812 01:04:34.529929 29216 fs_manager.cc:359] Metadata directory not provided
I20260812 01:04:34.530055 29216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:04:34.530396 29216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:04:34.533384 29216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-2/data/instance:
uuid: "627fecb70cc9491084c8b1fced0966d0"
format_stamp: "Formatted at 2026-08-12 01:04:34 on dist-test-slave-7f01"
server_key: "41504aea7f56bd49a930379ce2ad66ef"
server_key_iv: "d0f5e1fbc2700b0306731384b729efee"
server_key_version: "encryptionkey@0"
I20260812 01:04:34.534212 29216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-2/wal/instance:
uuid: "627fecb70cc9491084c8b1fced0966d0"
format_stamp: "Formatted at 2026-08-12 01:04:34 on dist-test-slave-7f01"
server_key: "41504aea7f56bd49a930379ce2ad66ef"
server_key_iv: "d0f5e1fbc2700b0306731384b729efee"
server_key_version: "encryptionkey@0"
I20260812 01:04:34.539321 29216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 01:04:34.542730 29244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:04:34.543987 29216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:04:34.544181 29216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-2/wal
uuid: "627fecb70cc9491084c8b1fced0966d0"
format_stamp: "Formatted at 2026-08-12 01:04:34 on dist-test-slave-7f01"
server_key: "41504aea7f56bd49a930379ce2ad66ef"
server_key_iv: "d0f5e1fbc2700b0306731384b729efee"
server_key_version: "encryptionkey@0"
I20260812 01:04:34.544420 29216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:04:34.575238 29216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:04:34.579113 29216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:04:34.579411 29216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:04:34.598610 29216 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.124:35273
I20260812 01:04:34.598647 29334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.124:35273 every 8 connection(s)
I20260812 01:04:34.599963 29216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-2/data/info.pb
I20260812 01:04:34.602653 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 29216
I20260812 01:04:34.602861 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786496278741276-22309-0/minicluster-data/master-2/wal/instance
I20260812 01:04:34.603313 22309 external_mini_cluster.cc:1428] Setting key 6b7a60c0557c9763831a1db6c8874cc5
I20260812 01:04:34.603678 29335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:04:34.613380 29335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41499 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:04:34 dist-test-slave-7f01 krb5kdc[28953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.124@KRBTEST.COM
Aug 12 01:04:34 dist-test-slave-7f01 krb5kdc[28953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496674, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.124@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:04:34.628180 29339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.605314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34344 (local address 127.21.201.124:35273)
0812 01:04:34.606176 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:34.606187 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:34.606228 (+ 41us) server_negotiation.cc:407] Connection header received
0812 01:04:34.608303 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:34.608324 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:34.608694 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:34.609004 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:34.610925 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.612302 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.614176 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.614929 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.615065 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:34.619897 (+ 4832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:34.619910 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:34.619937 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:34.619984 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:34.623393 (+ 3409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.624288 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.624299 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.624320 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.624541 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.624985 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.624990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.624992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.625518 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:34.625867 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:34.626234 (+ 367us) server_negotiation.cc:299] Negotiation successful
0812 01:04:34.626662 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":399,"threads_started":1}
I20260812 01:04:34.632622 29342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.616325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:41499 (local address 127.0.0.1:39790)
0812 01:04:34.617239 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:34.617276 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 01:04:34.617469 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:04:34.617898 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:04:34.617906 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:04:34.618516 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:04:34.618774 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:34.618783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.619938 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.619942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:34.621053 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:34.621065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.622105 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.622110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:34.622242 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:34.622864 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:04:34.622880 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:04:34.625601 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:04:34.629580 (+ 3979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:34.629589 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:34.629598 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:34.630264 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:34.630671 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:34.630674 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:34.630676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:34.630852 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:34.631534 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:04:34.631541 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:04:34.631775 (+ 234us) client_negotiation.cc:769] Sending connection context
0812 01:04:34.632056 (+ 281us) client_negotiation.cc:240] Negotiation successful
0812 01:04:34.632311 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":586,"threads_started":1}
I20260812 01:04:34.632897 29341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.616356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39790 (local address 127.21.201.126:41499)
0812 01:04:34.616907 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:34.616916 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:34.617388 (+ 472us) server_negotiation.cc:407] Connection header received
0812 01:04:34.617629 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:34.617634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:34.617702 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:34.617822 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:34.618986 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.619736 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.621225 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.621916 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.622051 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:34.625833 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:34.625839 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:34.625844 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:34.625901 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:34.629424 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.630391 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.630395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.630397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.630533 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.630996 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.631004 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.631007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.631370 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:34.631548 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:34.632171 (+ 623us) server_negotiation.cc:299] Negotiation successful
0812 01:04:34.632407 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":274,"threads_started":1}
I20260812 01:04:34.634542 29335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:04:34 dist-test-slave-7f01 krb5kdc[28953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
Aug 12 01:04:34 dist-test-slave-7f01 krb5kdc[28953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496674, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.124@KRBTEST.COM for kudu/127.21.201.125@KRBTEST.COM
I20260812 01:04:34.647852 29348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.631677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39792 (local address 127.21.201.126:41499)
0812 01:04:34.632378 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:34.632384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:34.632407 (+ 23us) server_negotiation.cc:407] Connection header received
0812 01:04:34.632453 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:34.632457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:34.632522 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:34.632636 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:34.634099 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.635140 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.636807 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.637248 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.637391 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:34.641764 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:34.641773 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:34.641781 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:34.641819 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:34.644864 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.645660 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.645665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.645668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.645826 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.646194 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.646201 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.646206 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.646545 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:34.646854 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:34.647151 (+ 297us) server_negotiation.cc:299] Negotiation successful
0812 01:04:34.647374 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":291,"threads_started":1}
I20260812 01:04:34.648356 29342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.635328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:44351 (local address 127.0.0.1:53640)
0812 01:04:34.635520 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:34.635539 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:04:34.635659 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:04:34.635996 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:04:34.636001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:04:34.636395 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:04:34.636521 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:34.636529 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.637488 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.637493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:34.638532 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:34.638545 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.639276 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.639282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:34.639481 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:34.640113 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:04:34.640129 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:04:34.642287 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:04:34.645613 (+ 3326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:34.645621 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:34.645628 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:34.646138 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:34.646521 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:34.646527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:34.646530 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:34.646703 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:34.647346 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:04:34.647351 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:04:34.647526 (+ 175us) client_negotiation.cc:769] Sending connection context
0812 01:04:34.647689 (+ 163us) client_negotiation.cc:240] Negotiation successful
0812 01:04:34.648030 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260812 01:04:34.648803 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.635371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53640 (local address 127.21.201.125:44351)
0812 01:04:34.635598 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:34.635604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:34.635623 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:04:34.635780 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:34.635787 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:34.635861 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:34.635968 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:34.636671 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.637350 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.638682 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.639119 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.639261 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:34.642516 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:34.642521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:34.642524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:34.642557 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:34.645459 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.646252 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.646256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.646258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.646385 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.646836 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.646841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.646844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.647210 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:34.647359 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:34.647865 (+ 506us) server_negotiation.cc:299] Negotiation successful
0812 01:04:34.648140 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 01:04:34.649895 29335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:04:34 dist-test-slave-7f01 krb5kdc[28953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496674, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.124@KRBTEST.COM for kudu/127.21.201.124@KRBTEST.COM
Aug 12 01:04:34 dist-test-slave-7f01 krb5kdc[28953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.125@KRBTEST.COM
I20260812 01:04:34.658638 29094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } attempt: 4
I20260812 01:04:34.664583 29342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.650781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:35273 (local address 127.0.0.1:34360)
0812 01:04:34.650937 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:34.650955 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:04:34.651053 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:04:34.651324 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:04:34.651327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:04:34.651644 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:04:34.651840 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:34.651850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.652853 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.652858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:34.653761 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:34.653773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.654593 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.654601 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:34.654827 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:34.655452 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:04:34.655468 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:04:34.657709 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:04:34.661617 (+ 3908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:34.661626 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:34.661631 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:34.662196 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:34.662633 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:34.662638 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:34.662644 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:34.662825 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:34.663495 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:04:34.663500 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:04:34.663635 (+ 135us) client_negotiation.cc:769] Sending connection context
0812 01:04:34.663843 (+ 208us) client_negotiation.cc:240] Negotiation successful
0812 01:04:34.664148 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260812 01:04:34.664619 29339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.650686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34360 (local address 127.21.201.124:35273)
0812 01:04:34.650907 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:34.650913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:34.651039 (+ 126us) server_negotiation.cc:407] Connection header received
0812 01:04:34.651133 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:34.651139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:34.651213 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:34.651325 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:34.651987 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.652711 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.653909 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.654403 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.654505 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:34.657892 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:34.657897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:34.657900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:34.657930 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:34.661471 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.662331 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.662337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.662340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.662490 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.662960 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.662965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.662968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.663340 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:34.663529 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:34.663950 (+ 421us) server_negotiation.cc:299] Negotiation successful
0812 01:04:34.664292 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":61}
I20260812 01:04:34.666375 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.649931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53652 (local address 127.21.201.125:44351)
0812 01:04:34.650137 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:34.650142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:34.650198 (+ 56us) server_negotiation.cc:407] Connection header received
0812 01:04:34.650381 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:34.650384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:34.650463 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:34.650567 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:34.652119 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.652833 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.654066 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.654536 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.654662 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:34.659621 (+ 4959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:34.659628 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:34.659633 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:34.659673 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:34.663133 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.664096 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.664101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.664105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.664323 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.664793 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.664798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.664802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.665170 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:34.665482 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:34.665772 (+ 290us) server_negotiation.cc:299] Negotiation successful
0812 01:04:34.666026 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Aug 12 01:04:34 dist-test-slave-7f01 krb5kdc[28953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496673, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for kudu/127.21.201.124@KRBTEST.COM
I20260812 01:04:34.672180 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:04:34.672395 29335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 627fecb70cc9491084c8b1fced0966d0: Bootstrap starting.
I20260812 01:04:34.675815 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.659336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:35273 (local address 127.0.0.1:34374)
0812 01:04:34.659610 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:34.659627 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 01:04:34.659725 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:04:34.661237 (+ 1512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:04:34.661243 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:04:34.661744 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:04:34.661927 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:34.661937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.663425 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.663433 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:34.664989 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:34.665004 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.666013 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.666019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:34.666236 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:34.666806 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:04:34.666823 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:04:34.669296 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:04:34.673263 (+ 3967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:34.673268 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:34.673272 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:34.673659 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:34.674065 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:34.674071 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:34.674074 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:34.674217 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:34.674759 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:04:34.674764 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:04:34.674928 (+ 164us) client_negotiation.cc:769] Sending connection context
0812 01:04:34.675073 (+ 145us) client_negotiation.cc:240] Negotiation successful
0812 01:04:34.675352 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260812 01:04:34.676182 29349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.659382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34374 (local address 127.21.201.124:35273)
0812 01:04:34.660891 (+ 1509us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:34.660896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:34.660921 (+ 25us) server_negotiation.cc:407] Connection header received
0812 01:04:34.660972 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:34.660977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:34.661051 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:34.661168 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:34.662102 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.663280 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.665134 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.665815 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.666542 (+ 727us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:34.669521 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:34.669526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:34.669529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:34.669594 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:34.673120 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.673788 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.673793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.673796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.673943 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.674335 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.674340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.674343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.674633 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:34.674793 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:34.675174 (+ 381us) server_negotiation.cc:299] Negotiation successful
0812 01:04:34.675358 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1358,"thread_start_us":987,"threads_started":1}
I20260812 01:04:34.676966 29335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 627fecb70cc9491084c8b1fced0966d0: Neither blocks nor log segments found. Creating new log.
I20260812 01:04:34.678387 29335 log.cc:824] T 00000000000000000000000000000000 P 627fecb70cc9491084c8b1fced0966d0: Log is configured to *not* fsync() on all Append() calls
I20260812 01:04:34.675843 29043 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "875f42f7d1c74e89beae9721c18e389e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39792
I20260812 01:04:34.681213 29094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e: Bootstrap starting.
I20260812 01:04:34.681751 29335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 627fecb70cc9491084c8b1fced0966d0: No bootstrap required, opened a new log
I20260812 01:04:34.684345 29094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e: Neither blocks nor log segments found. Creating new log.
I20260812 01:04:34.685338 29094 log.cc:824] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e: Log is configured to *not* fsync() on all Append() calls
I20260812 01:04:34.688704 29094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e: No bootstrap required, opened a new log
I20260812 01:04:34.690616 29335 raft_consensus.cc:348] T 00000000000000000000000000000000 P 627fecb70cc9491084c8b1fced0966d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f42f7d1c74e89beae9721c18e389e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41499 } } peers { permanent_uuid: "ee5626ae096042e187f7e7e860b0228f" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 } } peers { permanent_uuid: "627fecb70cc9491084c8b1fced0966d0" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } }
I20260812 01:04:34.690968 29335 raft_consensus.cc:729] T 00000000000000000000000000000000 P 627fecb70cc9491084c8b1fced0966d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 627fecb70cc9491084c8b1fced0966d0, State: Initialized, Role: FOLLOWER
I20260812 01:04:34.691392 29335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 627fecb70cc9491084c8b1fced0966d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f42f7d1c74e89beae9721c18e389e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41499 } } peers { permanent_uuid: "ee5626ae096042e187f7e7e860b0228f" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 } } peers { permanent_uuid: "627fecb70cc9491084c8b1fced0966d0" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } }
I20260812 01:04:34.692334 29355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 627fecb70cc9491084c8b1fced0966d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f42f7d1c74e89beae9721c18e389e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41499 } } peers { permanent_uuid: "ee5626ae096042e187f7e7e860b0228f" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 } } peers { permanent_uuid: "627fecb70cc9491084c8b1fced0966d0" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } } }
I20260812 01:04:34.692757 29355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 627fecb70cc9491084c8b1fced0966d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:04:34.693497 29335 sys_catalog.cc:564] T 00000000000000000000000000000000 P 627fecb70cc9491084c8b1fced0966d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:04:34.696221 29094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f42f7d1c74e89beae9721c18e389e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41499 } } peers { permanent_uuid: "ee5626ae096042e187f7e7e860b0228f" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 } } peers { permanent_uuid: "627fecb70cc9491084c8b1fced0966d0" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } }
I20260812 01:04:34.696611 29094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 875f42f7d1c74e89beae9721c18e389e, State: Initialized, Role: FOLLOWER
I20260812 01:04:34.697072 29094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f42f7d1c74e89beae9721c18e389e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41499 } } peers { permanent_uuid: "ee5626ae096042e187f7e7e860b0228f" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 } } peers { permanent_uuid: "627fecb70cc9491084c8b1fced0966d0" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } }
I20260812 01:04:34.698113 29371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f42f7d1c74e89beae9721c18e389e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41499 } } peers { permanent_uuid: "ee5626ae096042e187f7e7e860b0228f" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 } } peers { permanent_uuid: "627fecb70cc9491084c8b1fced0966d0" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } } }
I20260812 01:04:34.698535 29371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:04:34.699155 29094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:04:34.699365 29043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 01:04:34.699599 29043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:04:34.704483 29043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f42f7d1c74e89beae9721c18e389e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41499 } } peers { permanent_uuid: "ee5626ae096042e187f7e7e860b0228f" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 } } peers { permanent_uuid: "627fecb70cc9491084c8b1fced0966d0" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } }
I20260812 01:04:34.706601 29043 leader_election.cc:288] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e [CANDIDATE]: Term 1 election: Requested vote from peers ee5626ae096042e187f7e7e860b0228f (127.21.201.125:44351), 627fecb70cc9491084c8b1fced0966d0 (127.21.201.124:35273)
I20260812 01:04:34.709164 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "875f42f7d1c74e89beae9721c18e389e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "627fecb70cc9491084c8b1fced0966d0"
I20260812 01:04:34.709707 29293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 627fecb70cc9491084c8b1fced0966d0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:04:34.718822 29293 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 627fecb70cc9491084c8b1fced0966d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 875f42f7d1c74e89beae9721c18e389e in term 1.
I20260812 01:04:34.719632 28999 leader_election.cc:302] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 627fecb70cc9491084c8b1fced0966d0, 875f42f7d1c74e89beae9721c18e389e; no voters:
I20260812 01:04:34.720264 29371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:04:34.739238 29371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e [term 1 LEADER]: Becoming Leader. State: Replica: 875f42f7d1c74e89beae9721c18e389e, State: Running, Role: LEADER
I20260812 01:04:34.740306 29371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f42f7d1c74e89beae9721c18e389e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41499 } } peers { permanent_uuid: "ee5626ae096042e187f7e7e860b0228f" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 } } peers { permanent_uuid: "627fecb70cc9491084c8b1fced0966d0" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } }
W20260812 01:04:34.741043 29391 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 627fecb70cc9491084c8b1fced0966d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 01:04:34.741379 29391 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 01:04:34.713346 29216 master_runner.cc:186] Error getting master registration for 127.21.201.126:41499: OK, instance_id { permanent_uuid: "875f42f7d1c74e89beae9721c18e389e" instance_seqno: 1786496673509943 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 01:04:34.748286 29216 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 01:04:34.748445 29005 tablet.cc:2369] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:04:34.753068 29404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 875f42f7d1c74e89beae9721c18e389e. Latest consensus state: current_term: 1 leader_uuid: "875f42f7d1c74e89beae9721c18e389e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f42f7d1c74e89beae9721c18e389e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41499 } } peers { permanent_uuid: "ee5626ae096042e187f7e7e860b0228f" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 } } peers { permanent_uuid: "627fecb70cc9491084c8b1fced0966d0" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } } }
I20260812 01:04:34.753595 29404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:34.772212 29348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.744165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39798 (local address 127.21.201.126:41499)
0812 01:04:34.744421 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:34.744430 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:34.749024 (+ 4594us) server_negotiation.cc:407] Connection header received
0812 01:04:34.749346 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:34.749353 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:34.749444 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:34.749530 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:34.754712 (+ 5182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.756286 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.758989 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.759675 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.759888 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:34.763707 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:34.763713 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:34.763718 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:34.763778 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:34.768523 (+ 4745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.769501 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.769507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.769510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.769690 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.770181 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.770188 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.770192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.770581 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:34.770851 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:34.771510 (+ 659us) server_negotiation.cc:299] Negotiation successful
0812 01:04:34.771779 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 01:04:34.759511 28974 master_runner.cc:186] Error getting master registration for 127.21.201.125:44351: OK, instance_id { permanent_uuid: "ee5626ae096042e187f7e7e860b0228f" instance_seqno: 1786496674050616 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 01:04:34.775537 28974 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 01:04:34.779820 29204 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } attempt: 5
W20260812 01:04:34.782933 29395 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 01:04:34.783339 29395 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 01:04:34.784278 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.749416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53666 (local address 127.21.201.125:44351)
0812 01:04:34.749652 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:34.749660 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:34.754268 (+ 4608us) server_negotiation.cc:407] Connection header received
0812 01:04:34.754691 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:34.754697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:34.754787 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:34.754866 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:34.759162 (+ 4296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.760324 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.764939 (+ 4615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.766181 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.766304 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:34.776147 (+ 9843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:34.776155 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:34.776160 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:34.776211 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:34.780067 (+ 3856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.781703 (+ 1636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.781709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.781712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.781930 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.782401 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.782406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.782410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.782777 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:34.783037 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:34.783581 (+ 544us) server_negotiation.cc:299] Negotiation successful
0812 01:04:34.783902 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 01:04:34.787410 29414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:04:34.789842 29414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 01:04:34 dist-test-slave-7f01 krb5kdc[28953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496673, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.125@KRBTEST.COM for kudu/127.21.201.124@KRBTEST.COM
I20260812 01:04:34.801191 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.780373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:35273 (local address 127.0.0.1:34386)
0812 01:04:34.780626 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:34.780645 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 01:04:34.780747 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:04:34.781265 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:04:34.781270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:04:34.781802 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:04:34.782026 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:34.782039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.783684 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.783689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:34.785227 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:34.785243 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.786882 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.786889 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:34.787137 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:34.790946 (+ 3809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:04:34.790969 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:04:34.793866 (+ 2897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:04:34.798116 (+ 4250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:34.798121 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:34.798125 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:34.798610 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:34.799079 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:34.799085 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:34.799089 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:34.799257 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:34.800068 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:04:34.800074 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:04:34.800294 (+ 220us) client_negotiation.cc:769] Sending connection context
0812 01:04:34.800459 (+ 165us) client_negotiation.cc:240] Negotiation successful
0812 01:04:34.800795 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260812 01:04:34.802115 29339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.780470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34386 (local address 127.21.201.124:35273)
0812 01:04:34.780669 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:34.780677 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:34.780698 (+ 21us) server_negotiation.cc:407] Connection header received
0812 01:04:34.780881 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:34.780886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:34.780967 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:34.781063 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:34.782207 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.783348 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.785730 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.786412 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.786579 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:34.794089 (+ 7510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:34.794098 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:34.794109 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:34.794149 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:34.797929 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.798769 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.798775 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.798778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.798943 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.799405 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.799410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.799414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.799842 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:34.800000 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:34.801508 (+ 1508us) server_negotiation.cc:299] Negotiation successful
0812 01:04:34.801729 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 01:04:34.806267 29349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:34.753458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34380 (local address 127.21.201.124:35273)
0812 01:04:34.754073 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:34.754083 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:34.764462 (+ 10379us) server_negotiation.cc:407] Connection header received
0812 01:04:34.768132 (+ 3670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:34.768139 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:34.768243 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:34.768324 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:34.773014 (+ 4690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.774097 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.776018 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:34.776556 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:34.776662 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:34.786337 (+ 9675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:34.786344 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:34.786349 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:34.786392 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:34.790670 (+ 4278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.803770 (+ 13100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.803775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.803778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.803947 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:34.804434 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:34.804439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:34.804443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:34.804790 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:34.805042 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:34.805666 (+ 624us) server_negotiation.cc:299] Negotiation successful
0812 01:04:34.805913 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488}
I20260812 01:04:34.812963 29204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee5626ae096042e187f7e7e860b0228f: Bootstrap starting.
I20260812 01:04:34.818583 29204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee5626ae096042e187f7e7e860b0228f: Neither blocks nor log segments found. Creating new log.
I20260812 01:04:34.821882 29293 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 627fecb70cc9491084c8b1fced0966d0 [term 1 FOLLOWER]: Refusing update from remote peer 875f42f7d1c74e89beae9721c18e389e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:04:34.821429 29204 log.cc:824] T 00000000000000000000000000000000 P ee5626ae096042e187f7e7e860b0228f: Log is configured to *not* fsync() on all Append() calls
I20260812 01:04:34.823144 29404 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e [LEADER]: Connected to new peer: Peer: permanent_uuid: "627fecb70cc9491084c8b1fced0966d0" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 01:04:34.831420 29254 tablet.cc:2369] T 00000000000000000000000000000000 P 627fecb70cc9491084c8b1fced0966d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:04:34.837329 29204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee5626ae096042e187f7e7e860b0228f: No bootstrap required, opened a new log
I20260812 01:04:34.850975 29355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 627fecb70cc9491084c8b1fced0966d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 875f42f7d1c74e89beae9721c18e389e. Latest consensus state: current_term: 1 leader_uuid: "875f42f7d1c74e89beae9721c18e389e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f42f7d1c74e89beae9721c18e389e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41499 } } peers { permanent_uuid: "ee5626ae096042e187f7e7e860b0228f" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 } } peers { permanent_uuid: "627fecb70cc9491084c8b1fced0966d0" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } } }
I20260812 01:04:34.851336 29355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 627fecb70cc9491084c8b1fced0966d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:04:34.856009 29204 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee5626ae096042e187f7e7e860b0228f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f42f7d1c74e89beae9721c18e389e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41499 } } peers { permanent_uuid: "ee5626ae096042e187f7e7e860b0228f" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 } } peers { permanent_uuid: "627fecb70cc9491084c8b1fced0966d0" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } }
I20260812 01:04:34.856433 29204 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee5626ae096042e187f7e7e860b0228f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee5626ae096042e187f7e7e860b0228f, State: Initialized, Role: FOLLOWER
I20260812 01:04:34.856973 29204 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee5626ae096042e187f7e7e860b0228f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f42f7d1c74e89beae9721c18e389e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41499 } } peers { permanent_uuid: "ee5626ae096042e187f7e7e860b0228f" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 } } peers { permanent_uuid: "627fecb70cc9491084c8b1fced0966d0" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } }
I20260812 01:04:34.858291 29424 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee5626ae096042e187f7e7e860b0228f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f42f7d1c74e89beae9721c18e389e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41499 } } peers { permanent_uuid: "ee5626ae096042e187f7e7e860b0228f" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 } } peers { permanent_uuid: "627fecb70cc9491084c8b1fced0966d0" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } } }
I20260812 01:04:34.861418 29418 mvcc.cc:205] Tried to move back new op lower bound from 7317490380039127040 to 7317490379742220288. Current Snapshot: MvccSnapshot[applied={T|T < 7317490380039127040}]
I20260812 01:04:34.859143 29424 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee5626ae096042e187f7e7e860b0228f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:04:34.863169 29355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 627fecb70cc9491084c8b1fced0966d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "875f42f7d1c74e89beae9721c18e389e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f42f7d1c74e89beae9721c18e389e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41499 } } peers { permanent_uuid: "ee5626ae096042e187f7e7e860b0228f" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 } } peers { permanent_uuid: "627fecb70cc9491084c8b1fced0966d0" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } } }
I20260812 01:04:34.863399 29204 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee5626ae096042e187f7e7e860b0228f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:04:34.863409 29355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 627fecb70cc9491084c8b1fced0966d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:04:34.864007 29164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee5626ae096042e187f7e7e860b0228f [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:04:34.865339 29404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "875f42f7d1c74e89beae9721c18e389e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f42f7d1c74e89beae9721c18e389e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41499 } } peers { permanent_uuid: "ee5626ae096042e187f7e7e860b0228f" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 } } peers { permanent_uuid: "627fecb70cc9491084c8b1fced0966d0" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } } }
I20260812 01:04:34.865636 29404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:34.867264 29165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "875f42f7d1c74e89beae9721c18e389e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ee5626ae096042e187f7e7e860b0228f"
I20260812 01:04:34.867970 29414 catalog_manager.cc:1347] Generated new cluster ID: fedefc8ebc694a988ea750a5ad85a75c
I20260812 01:04:34.868095 29414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:04:34.872463 29164 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee5626ae096042e187f7e7e860b0228f [term 1 FOLLOWER]: Refusing update from remote peer 875f42f7d1c74e89beae9721c18e389e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:04:34.880081 29165 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ee5626ae096042e187f7e7e860b0228f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 875f42f7d1c74e89beae9721c18e389e in term 1.
I20260812 01:04:34.883133 29371 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee5626ae096042e187f7e7e860b0228f" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:04:34.937062 29414 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:04:34.939328 29414 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:04:34.946566 29424 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee5626ae096042e187f7e7e860b0228f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 875f42f7d1c74e89beae9721c18e389e. Latest consensus state: current_term: 1 leader_uuid: "875f42f7d1c74e89beae9721c18e389e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f42f7d1c74e89beae9721c18e389e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41499 } } peers { permanent_uuid: "ee5626ae096042e187f7e7e860b0228f" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 } } peers { permanent_uuid: "627fecb70cc9491084c8b1fced0966d0" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } } }
I20260812 01:04:34.947067 29424 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee5626ae096042e187f7e7e860b0228f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:04:34.952476 29439 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee5626ae096042e187f7e7e860b0228f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "875f42f7d1c74e89beae9721c18e389e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f42f7d1c74e89beae9721c18e389e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41499 } } peers { permanent_uuid: "ee5626ae096042e187f7e7e860b0228f" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 } } peers { permanent_uuid: "627fecb70cc9491084c8b1fced0966d0" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } } }
I20260812 01:04:34.952953 29439 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee5626ae096042e187f7e7e860b0228f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:04:34.953075 29371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "875f42f7d1c74e89beae9721c18e389e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f42f7d1c74e89beae9721c18e389e" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 41499 } } peers { permanent_uuid: "ee5626ae096042e187f7e7e860b0228f" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 44351 } } peers { permanent_uuid: "627fecb70cc9491084c8b1fced0966d0" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 35273 } } }
I20260812 01:04:34.955453 29371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:34.974679 29414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 875f42f7d1c74e89beae9721c18e389e: Generated new TSK 0
I20260812 01:04:34.976307 29414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 01:04:35.015889 29458 catalog_manager.cc:1259] Loaded cluster ID: fedefc8ebc694a988ea750a5ad85a75c
I20260812 01:04:35.016078 29458 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ee5626ae096042e187f7e7e860b0228f: loading cluster ID for follower catalog manager: success
I20260812 01:04:35.020102 29458 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ee5626ae096042e187f7e7e860b0228f: acquiring CA information for follower catalog manager: success
I20260812 01:04:35.022823 29458 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ee5626ae096042e187f7e7e860b0228f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 01:04:35.130915 29348 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:04:35.118128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39804 (local address 127.21.201.126:41499)
0812 01:04:35.118357 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:35.118366 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:35.128303 (+ 9937us) server_negotiation.cc:407] Connection header received
0812 01:04:35.128665 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:35.128674 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:35.128786 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:35.128884 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:35.130510 (+ 1626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39804: BlockingRecv error: recv got EOF from 127.0.0.1:39804 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260812 01:04:35.139295 29213 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:04:35.128516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53678 (local address 127.21.201.125:44351)
0812 01:04:35.128724 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:35.128730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:35.136977 (+ 8247us) server_negotiation.cc:407] Connection header received
0812 01:04:35.137687 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:35.137694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:35.137789 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:35.137878 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:35.139025 (+ 1147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53678: BlockingRecv error: recv got EOF from 127.0.0.1:53678 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260812 01:04:35.142149 29349 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:04:35.122744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34388 (local address 127.21.201.124:35273)
0812 01:04:35.122944 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:35.122951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:35.133611 (+ 10660us) server_negotiation.cc:407] Connection header received
0812 01:04:35.140080 (+ 6469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:35.140090 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:35.140206 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:35.140298 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:35.141794 (+ 1496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34388: BlockingRecv error: recv got EOF from 127.0.0.1:34388 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 01:04:35.190817 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:35.175369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53684 (local address 127.21.201.125:44351)
0812 01:04:35.175567 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:35.175574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:35.182693 (+ 7119us) server_negotiation.cc:407] Connection header received
0812 01:04:35.183219 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:35.183227 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:35.183460 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:35.183588 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:04:35.184364 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:35.185527 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:35.187614 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:35.188333 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:35.188449 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:35.189252 (+ 803us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:04:35.189934 (+ 682us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:04:35.190064 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:35.190314 (+ 250us) server_negotiation.cc:299] Negotiation successful
0812 01:04:35.190423 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260812 01:04:35.238371 29349 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:04:35.223107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34400 (local address 127.21.201.124:35273)
0812 01:04:35.223337 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:35.223346 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:35.236393 (+ 13047us) server_negotiation.cc:407] Connection header received
0812 01:04:35.236676 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:35.236688 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:35.236837 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:35.236952 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:35.238105 (+ 1153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34400: BlockingRecv error: recv got EOF from 127.0.0.1:34400 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 01:04:35.288812 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:35.274271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53694 (local address 127.21.201.125:44351)
0812 01:04:35.274487 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:35.274495 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:35.281410 (+ 6915us) server_negotiation.cc:407] Connection header received
0812 01:04:35.281967 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:35.281974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:35.282076 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:35.282147 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:04:35.282893 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:35.284099 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:35.286215 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:35.287015 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:35.287117 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:35.287867 (+ 750us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:04:35.288056 (+ 189us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:04:35.288125 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:35.288370 (+ 245us) server_negotiation.cc:299] Negotiation successful
0812 01:04:35.288480 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260812 01:04:35.334602 29349 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:04:35.324569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34402 (local address 127.21.201.124:35273)
0812 01:04:35.324786 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:35.324793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:35.332283 (+ 7490us) server_negotiation.cc:407] Connection header received
0812 01:04:35.332884 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:35.332891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:35.333001 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:35.333102 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:35.334311 (+ 1209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34402: BlockingRecv error: recv got EOF from 127.0.0.1:34402 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 01:04:35.384395 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:35.369689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53708 (local address 127.21.201.125:44351)
0812 01:04:35.369932 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:35.369939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:35.376965 (+ 7026us) server_negotiation.cc:407] Connection header received
0812 01:04:35.377518 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:35.377526 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:35.377625 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:35.377698 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:04:35.378490 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:35.379723 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:35.381869 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:35.382580 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:35.382697 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:35.383354 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:04:35.383532 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:04:35.383613 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:35.383927 (+ 314us) server_negotiation.cc:299] Negotiation successful
0812 01:04:35.384028 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260812 01:04:35.401871 29349 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:04:35.399577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34410 (local address 127.21.201.124:35273)
0812 01:04:35.399802 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:35.399808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:35.399940 (+ 132us) server_negotiation.cc:407] Connection header received
0812 01:04:35.400315 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:35.400322 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:35.400438 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:35.400526 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:35.401606 (+ 1080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34410: BlockingRecv error: recv got EOF from 127.0.0.1:34410 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 01:04:35.418203 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:35.410892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53718 (local address 127.21.201.125:44351)
0812 01:04:35.411146 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:35.411152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:35.411543 (+ 391us) server_negotiation.cc:407] Connection header received
0812 01:04:35.411728 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:35.411734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:35.411862 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:35.411982 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:04:35.412489 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:35.413790 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:35.415879 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:35.416625 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:35.416762 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:35.417328 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:04:35.417483 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:04:35.417591 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:35.417790 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 01:04:35.417878 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
W20260812 01:04:35.428234 29349 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:04:35.426016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34426 (local address 127.21.201.124:35273)
0812 01:04:35.426285 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:35.426293 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:35.426313 (+ 20us) server_negotiation.cc:407] Connection header received
0812 01:04:35.426556 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:35.426564 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:35.426676 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:35.426795 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:35.427970 (+ 1175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34426: BlockingRecv error: recv got EOF from 127.0.0.1:34426 (error 108)
Metrics: {"server-negotiator.queue_time_us":155}
I20260812 01:04:35.447600 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:35.440185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53730 (local address 127.21.201.125:44351)
0812 01:04:35.440345 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:35.440351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:35.440857 (+ 506us) server_negotiation.cc:407] Connection header received
0812 01:04:35.441027 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:35.441032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:35.441115 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:35.441210 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:04:35.441827 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:35.442989 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:35.445039 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:35.445788 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:35.445954 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:35.446388 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:04:35.446583 (+ 195us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:04:35.446682 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:35.446862 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 01:04:35.446953 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260812 01:04:35.456802 29349 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:04:35.454537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34438 (local address 127.21.201.124:35273)
0812 01:04:35.454708 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:35.454714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:35.455237 (+ 523us) server_negotiation.cc:407] Connection header received
0812 01:04:35.455400 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:35.455406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:35.455526 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:35.455653 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:35.456558 (+ 905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34438: BlockingRecv error: recv got EOF from 127.0.0.1:34438 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 01:04:35.484376 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:35.477528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53744 (local address 127.21.201.125:44351)
0812 01:04:35.477706 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:35.477713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:35.478258 (+ 545us) server_negotiation.cc:407] Connection header received
0812 01:04:35.478464 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:35.478470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:35.478575 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:35.478683 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:04:35.479186 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:35.480501 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:35.482197 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:35.482852 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:35.483007 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:35.483497 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:04:35.483640 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:04:35.483770 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:35.483972 (+ 202us) server_negotiation.cc:299] Negotiation successful
0812 01:04:35.484072 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260812 01:04:35.493198 29349 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:04:35.491152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34446 (local address 127.21.201.124:35273)
0812 01:04:35.491338 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:35.491344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:35.491705 (+ 361us) server_negotiation.cc:407] Connection header received
0812 01:04:35.491949 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:35.491956 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:35.492048 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:35.492152 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:35.492922 (+ 770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34446: BlockingRecv error: recv got EOF from 127.0.0.1:34446 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 01:04:35.534296 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:35.528785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53752 (local address 127.21.201.125:44351)
0812 01:04:35.529020 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:35.529026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:35.529552 (+ 526us) server_negotiation.cc:407] Connection header received
0812 01:04:35.529745 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:35.529751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:35.529906 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:35.530004 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:04:35.530467 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:35.531479 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:35.532832 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:35.533257 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:35.533350 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:35.533698 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:04:35.533804 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:04:35.533874 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:35.534009 (+ 135us) server_negotiation.cc:299] Negotiation successful
0812 01:04:35.534071 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260812 01:04:35.541875 29349 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:04:35.539997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34458 (local address 127.21.201.124:35273)
0812 01:04:35.540267 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:35.540273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:35.540520 (+ 247us) server_negotiation.cc:407] Connection header received
0812 01:04:35.540652 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:35.540656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:35.540729 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:35.540811 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:35.541663 (+ 852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34458: BlockingRecv error: recv got EOF from 127.0.0.1:34458 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
I20260812 01:04:35.616199 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:35.609993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53754 (local address 127.21.201.125:44351)
0812 01:04:35.610228 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:35.610234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:35.610590 (+ 356us) server_negotiation.cc:407] Connection header received
0812 01:04:35.610860 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:35.610882 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:35.610974 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:35.611055 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:04:35.611568 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:35.612439 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:35.614378 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:35.614814 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:35.614907 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:35.615422 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:04:35.615534 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:04:35.615603 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:35.615819 (+ 216us) server_negotiation.cc:299] Negotiation successful
0812 01:04:35.615882 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260812 01:04:35.624351 29349 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:04:35.622212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34466 (local address 127.21.201.124:35273)
0812 01:04:35.622410 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:35.622416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:35.622989 (+ 573us) server_negotiation.cc:407] Connection header received
0812 01:04:35.623150 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:35.623155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:35.623228 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:35.623308 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:35.624132 (+ 824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34466: BlockingRecv error: recv got EOF from 127.0.0.1:34466 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 01:04:35.743894 29391 catalog_manager.cc:1259] Loaded cluster ID: fedefc8ebc694a988ea750a5ad85a75c
I20260812 01:04:35.744051 29391 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 627fecb70cc9491084c8b1fced0966d0: loading cluster ID for follower catalog manager: success
I20260812 01:04:35.746882 29391 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 627fecb70cc9491084c8b1fced0966d0: acquiring CA information for follower catalog manager: success
I20260812 01:04:35.748946 29391 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 627fecb70cc9491084c8b1fced0966d0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 01:04:35.761034 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:35.756024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53762 (local address 127.21.201.125:44351)
0812 01:04:35.756210 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:35.756215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:35.756601 (+ 386us) server_negotiation.cc:407] Connection header received
0812 01:04:35.756761 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:35.756765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:35.756858 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:35.756954 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:04:35.757330 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:35.758064 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:35.759468 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:35.759920 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:35.760023 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:35.760393 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:04:35.760516 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:04:35.760585 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:35.760720 (+ 135us) server_negotiation.cc:299] Negotiation successful
0812 01:04:35.760798 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 01:04:35.772277 29349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:35.766739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34470 (local address 127.21.201.124:35273)
0812 01:04:35.767002 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:35.767008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:35.767179 (+ 171us) server_negotiation.cc:407] Connection header received
0812 01:04:35.767328 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:35.767332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:35.767496 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:35.767594 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:04:35.768029 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:35.768770 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:35.770090 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:35.770597 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:35.770697 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:35.771138 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:04:35.771657 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:04:35.771829 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:35.771897 (+ 68us) server_negotiation.cc:299] Negotiation successful
0812 01:04:35.771966 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260812 01:04:35.788398 29349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:35.778620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34482 (local address 127.21.201.124:35273)
0812 01:04:35.778833 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:35.778842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:35.781549 (+ 2707us) server_negotiation.cc:407] Connection header received
0812 01:04:35.781768 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:35.781776 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:35.781880 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:35.781980 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:04:35.782817 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:35.784178 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:35.786198 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:35.786869 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:35.787020 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:35.787466 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:04:35.787659 (+ 193us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:04:35.787788 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:35.787969 (+ 181us) server_negotiation.cc:299] Negotiation successful
0812 01:04:35.788063 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 01:04:35.788959 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:35.778656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53778 (local address 127.21.201.125:44351)
0812 01:04:35.779444 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:35.779454 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:35.782374 (+ 2920us) server_negotiation.cc:407] Connection header received
0812 01:04:35.782735 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:35.782744 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:35.782848 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:35.782951 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:04:35.783623 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:35.784811 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:35.786712 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:35.787426 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:35.787593 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:35.788138 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:04:35.788265 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:04:35.788330 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:35.788583 (+ 253us) server_negotiation.cc:299] Negotiation successful
0812 01:04:35.788663 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605}
I20260812 01:04:35.790820 29641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:35.779064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39808 (local address 127.21.201.126:41499)
0812 01:04:35.781166 (+ 2102us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:35.781175 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:35.782782 (+ 1607us) server_negotiation.cc:407] Connection header received
0812 01:04:35.783141 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:35.783145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:35.783310 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:35.783437 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:04:35.784298 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:35.785230 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:35.787733 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:35.788384 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:35.788534 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:35.789115 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:04:35.789616 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:04:35.789728 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:35.790509 (+ 781us) server_negotiation.cc:299] Negotiation successful
0812 01:04:35.790605 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1929,"thread_start_us":1775,"threads_started":1}
I20260812 01:04:35.797014 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 28974
I20260812 01:04:35.798038 29087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:04:35.923575 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 28974
I20260812 01:04:35.954897 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 29104
I20260812 01:04:35.956122 29192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:04:36.070024 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 29104
I20260812 01:04:36.101133 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 29216
I20260812 01:04:36.102304 29328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:04:36.219599 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 29216
2026-08-12T01:04:36Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4871 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 01:04:36 dist-test-slave-7f01 krb5kdc[29651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 01:04:36 dist-test-slave-7f01 krb5kdc[29651](info): set up 2 sockets
Aug 12 01:04:36 dist-test-slave-7f01 krb5kdc[29651](info): commencing operation
krb5kdc: starting...
W20260812 01:04:37.599119 22309 mini_kdc.cc:121] Time spent starting KDC: real 1.331s user 0.000s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/krb5kdc/test-user.keytab.
Aug 12 01:04:37 dist-test-slave-7f01 krb5kdc[29651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T01:04:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T01:04:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.201.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
Entry for principal kudu/127.21.201.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab.
I20260812 01:04:37.928328 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:42797
--webserver_interface=127.21.201.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:43853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:42797,127.21.201.125:34241,127.21.201.124:38401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 01:04:38.275503 29672 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 01:04:38.275835 29672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:04:38.275913 29672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:04:38.275998 29672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:04:38.283130 29672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:04:38.283218 29672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:04:38.283277 29672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:04:38.283329 29672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:04:38.283390 29672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:04:38.293920 29672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:43853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:42797,127.21.201.125:34241,127.21.201.124:38401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.126.keytab
--principal=kudu/127.21.201.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.126:42797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.201.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:04:38.295161 29672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:04:38.296523 29672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:04:38.308473 29678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:38.308390 29682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:38.308439 29679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:04:38.309880 29672 server_base.cc:1034] running on GCE node
I20260812 01:04:38.310649 29672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:04:38.312083 29672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:04:38.313369 29672 hybrid_clock.cc:648] HybridClock initialized: now 1786496678313309 us; error 44 us; skew 500 ppm
I20260812 01:04:38.313740 29672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:04:38 dist-test-slave-7f01 krb5kdc[29651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496678, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:04:38.317065 29672 init.cc:377] Logged in from keytab as kudu/127.21.201.126@KRBTEST.COM (short username kudu)
I20260812 01:04:38.318624 29672 webserver.cc:460] Webserver started at http://127.21.201.126:42007/ using document root <none> and password file <none>
I20260812 01:04:38.319326 29672 fs_manager.cc:359] Metadata directory not provided
I20260812 01:04:38.319463 29672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:04:38.319862 29672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:04:38.322827 29672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-0/data/instance:
uuid: "98710605273c460790bc328105af2091"
format_stamp: "Formatted at 2026-08-12 01:04:38 on dist-test-slave-7f01"
server_key: "f33776a6ba7df64c0a7c03dec0660e36"
server_key_iv: "e499f212dc08b0718c89933153d5d0e6"
server_key_version: "encryptionkey@0"
I20260812 01:04:38.323668 29672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-0/wal/instance:
uuid: "98710605273c460790bc328105af2091"
format_stamp: "Formatted at 2026-08-12 01:04:38 on dist-test-slave-7f01"
server_key: "f33776a6ba7df64c0a7c03dec0660e36"
server_key_iv: "e499f212dc08b0718c89933153d5d0e6"
server_key_version: "encryptionkey@0"
I20260812 01:04:38.328799 29672 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 01:04:38.332263 29698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:04:38.333549 29672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 01:04:38.333745 29672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-0/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-0/wal
uuid: "98710605273c460790bc328105af2091"
format_stamp: "Formatted at 2026-08-12 01:04:38 on dist-test-slave-7f01"
server_key: "f33776a6ba7df64c0a7c03dec0660e36"
server_key_iv: "e499f212dc08b0718c89933153d5d0e6"
server_key_version: "encryptionkey@0"
I20260812 01:04:38.333968 29672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:04:38.358294 29672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:04:38.362244 29672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:04:38.362524 29672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:04:38.381799 29672 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.126:42797
I20260812 01:04:38.381839 29783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.126:42797 every 8 connection(s)
I20260812 01:04:38.383139 29672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-0/data/info.pb
I20260812 01:04:38.386713 29785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:04:38.389292 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 29672
I20260812 01:04:38.389493 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-0/wal/instance
I20260812 01:04:38.390036 22309 external_mini_cluster.cc:1428] Setting key d91d5c8c9057dc66205629f4ea4c241c
I20260812 01:04:38.395970 29785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42797 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:04:38 dist-test-slave-7f01 krb5kdc[29651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496678, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:04:38.418968 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:38.399857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:42797 (local address 127.0.0.1:42842)
0812 01:04:38.400740 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:38.400793 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 01:04:38.402788 (+ 1995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:04:38.403584 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:04:38.403590 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:04:38.404346 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:04:38.404686 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:38.404703 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:38.406273 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:38.406282 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:38.407331 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:38.407344 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:38.408337 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:38.408344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:38.408654 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:38.409354 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:04:38.409374 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:04:38.411837 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:04:38.414940 (+ 3103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:38.414951 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:38.414961 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:38.415346 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:38.415796 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:38.415801 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:38.415805 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:38.415957 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:38.416697 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:04:38.416704 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:04:38.416899 (+ 195us) client_negotiation.cc:769] Sending connection context
0812 01:04:38.417138 (+ 239us) client_negotiation.cc:240] Negotiation successful
0812 01:04:38.417697 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":204,"thread_start_us":393,"threads_started":1}
I20260812 01:04:38.419014 29789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:38.399853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42842 (local address 127.21.201.126:42797)
0812 01:04:38.401499 (+ 1646us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:38.401512 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:38.401564 (+ 52us) server_negotiation.cc:407] Connection header received
0812 01:04:38.403145 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:38.403151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:38.403363 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:38.403568 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:38.404903 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:38.406110 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:38.407473 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:38.408094 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:38.408292 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:38.412055 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:38.412063 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:38.412074 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:38.412106 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:38.414799 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:38.415496 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:38.415502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:38.415504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:38.415645 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:38.416080 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:38.416091 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:38.416093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:38.416548 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:38.416754 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:38.417276 (+ 522us) server_negotiation.cc:299] Negotiation successful
0812 01:04:38.417575 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":1428,"spinlock_wait_cycles":15232,"thread_start_us":1217,"threads_started":1}
I20260812 01:04:38.422379 29785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.201.125@KRBTEST.COM; defaulting to no policy
I20260812 01:04:38.423822 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:38.423217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:34241 (local address 127.0.0.1:44102)
0812 01:04:38.423389 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:38.423506 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260812 01:04:38.424144 29702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.201.125:34241: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.125@KRBTEST.COM" created.
I20260812 01:04:38.425967 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:38.425388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:34241 (local address 127.0.0.1:44106)
0812 01:04:38.425633 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:38.425750 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260812 01:04:38.426234 29785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.125:34241: Network error: Client connection negotiation failed: client connection to 127.21.201.125:34241: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.125.keytab.
Entry for principal kudu/127.21.201.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.125.keytab.
I20260812 01:04:38.462517 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.125:34241
--webserver_interface=127.21.201.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.125.keytab
--principal=kudu/127.21.201.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:43853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:42797,127.21.201.125:34241,127.21.201.124:38401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:04:38.475558 29785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 } attempt: 1
I20260812 01:04:38.476754 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:38.476278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:34241 (local address 127.0.0.1:44120)
0812 01:04:38.476482 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:38.476619 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260812 01:04:38.477819 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:38.477481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:34241 (local address 127.0.0.1:44136)
0812 01:04:38.477627 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:38.477702 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 01:04:38.478052 29785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.125:34241: Network error: Client connection negotiation failed: client connection to 127.21.201.125:34241: connect: Connection refused (error 111)
I20260812 01:04:38.546303 29785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 } attempt: 2
I20260812 01:04:38.547763 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:38.547150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:34241 (local address 127.0.0.1:44140)
0812 01:04:38.547482 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:38.547579 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260812 01:04:38.548872 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:38.548503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:34241 (local address 127.0.0.1:44154)
0812 01:04:38.548675 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:38.548746 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 01:04:38.549093 29785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.125:34241: Network error: Client connection negotiation failed: client connection to 127.21.201.125:34241: connect: Connection refused (error 111)
I20260812 01:04:38.640344 29785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 } attempt: 3
I20260812 01:04:38.641600 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:38.641131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:34241 (local address 127.0.0.1:44166)
0812 01:04:38.641359 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:38.641456 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260812 01:04:38.642753 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:38.642353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:34241 (local address 127.0.0.1:44176)
0812 01:04:38.642532 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:38.642616 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260812 01:04:38.642959 29785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.125:34241: Network error: Client connection negotiation failed: client connection to 127.21.201.125:34241: connect: Connection refused (error 111)
I20260812 01:04:38.786243 29785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 } attempt: 4
I20260812 01:04:38.787583 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:38.786984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:34241 (local address 127.0.0.1:44188)
0812 01:04:38.787342 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:38.787438 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260812 01:04:38.788693 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:38.788335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:34241 (local address 127.0.0.1:44200)
0812 01:04:38.788503 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:38.788579 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.125:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 01:04:38.788872 29785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.125:34241: Network error: Client connection negotiation failed: client connection to 127.21.201.125:34241: connect: Connection refused (error 111)
W20260812 01:04:38.823935 29794 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 01:04:38.824327 29794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:04:38.824460 29794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:04:38.824612 29794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:04:38.831935 29794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:04:38.832211 29794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:04:38.832370 29794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:04:38.832465 29794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:04:38.832577 29794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:04:38.842761 29794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:43853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:42797,127.21.201.125:34241,127.21.201.124:38401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.125.keytab
--principal=kudu/127.21.201.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.125:34241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.201.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:04:38.844154 29794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:04:38.845456 29794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:04:38.857537 29806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:38.857472 29813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:38.857666 29810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:04:38.858815 29794 server_base.cc:1034] running on GCE node
I20260812 01:04:38.859607 29794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:04:38.861068 29794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:04:38.862349 29794 hybrid_clock.cc:648] HybridClock initialized: now 1786496678862292 us; error 51 us; skew 500 ppm
I20260812 01:04:38.862701 29794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:04:38 dist-test-slave-7f01 krb5kdc[29651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496678, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:04:38.866776 29794 init.cc:377] Logged in from keytab as kudu/127.21.201.125@KRBTEST.COM (short username kudu)
I20260812 01:04:38.869575 29794 webserver.cc:460] Webserver started at http://127.21.201.125:38709/ using document root <none> and password file <none>
I20260812 01:04:38.870602 29794 fs_manager.cc:359] Metadata directory not provided
I20260812 01:04:38.870754 29794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:04:38.871186 29794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:04:38.876111 29794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-1/data/instance:
uuid: "2299e87e6e384819a6ef59d216aaa232"
format_stamp: "Formatted at 2026-08-12 01:04:38 on dist-test-slave-7f01"
server_key: "3af7b0eef01f6439268f928153403a9d"
server_key_iv: "d28e2cc889c1a5f01d67cff07e8c0859"
server_key_version: "encryptionkey@0"
I20260812 01:04:38.877449 29794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-1/wal/instance:
uuid: "2299e87e6e384819a6ef59d216aaa232"
format_stamp: "Formatted at 2026-08-12 01:04:38 on dist-test-slave-7f01"
server_key: "3af7b0eef01f6439268f928153403a9d"
server_key_iv: "d28e2cc889c1a5f01d67cff07e8c0859"
server_key_version: "encryptionkey@0"
I20260812 01:04:38.885895 29794 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.003s
I20260812 01:04:38.891371 29825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:04:38.893354 29794 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 01:04:38.893565 29794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-1/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-1/wal
uuid: "2299e87e6e384819a6ef59d216aaa232"
format_stamp: "Formatted at 2026-08-12 01:04:38 on dist-test-slave-7f01"
server_key: "3af7b0eef01f6439268f928153403a9d"
server_key_iv: "d28e2cc889c1a5f01d67cff07e8c0859"
server_key_version: "encryptionkey@0"
I20260812 01:04:38.893868 29794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:04:38.913141 29794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:04:38.918395 29794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:04:38.918680 29794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:04:38.939369 29794 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.125:34241
I20260812 01:04:38.939459 29909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.125:34241 every 8 connection(s)
I20260812 01:04:38.940794 29794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-1/data/info.pb
I20260812 01:04:38.944173 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 29794
I20260812 01:04:38.944381 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-1/wal/instance
I20260812 01:04:38.944842 22309 external_mini_cluster.cc:1428] Setting key 10dd9ac4da354e130ca5b8ab796a10b7
I20260812 01:04:38.945549 29910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:04:38.955449 29910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42797 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:04:38 dist-test-slave-7f01 krb5kdc[29651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496678, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.125@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.201.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.201.124@KRBTEST.COM" created.
I20260812 01:04:38.986755 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:38.959026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42852 (local address 127.21.201.126:42797)
0812 01:04:38.959612 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:38.959619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:38.963820 (+ 4201us) server_negotiation.cc:407] Connection header received
0812 01:04:38.966532 (+ 2712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:38.966539 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:38.966640 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:38.966801 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:38.969115 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:38.970290 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:38.972857 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:38.973565 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:38.973708 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:38.978787 (+ 5079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:38.978796 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:38.978803 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:38.978867 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:38.982961 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:38.984004 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:38.984008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:38.984011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:38.984141 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:38.984658 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:38.984661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:38.984663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:38.984947 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:38.985100 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:38.986076 (+ 976us) server_negotiation.cc:299] Negotiation successful
0812 01:04:38.986348 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":275,"threads_started":1}
I20260812 01:04:38.988348 29916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:38.959593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:42797 (local address 127.0.0.1:42852)
0812 01:04:38.963287 (+ 3694us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:38.963338 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 01:04:38.966187 (+ 2849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:04:38.967060 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:04:38.967073 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:04:38.968008 (+ 935us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:04:38.968829 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:38.968856 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:38.970484 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:38.970493 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:38.972683 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:38.972700 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:38.973834 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:38.973844 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:38.974224 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:38.975103 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:04:38.975145 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:04:38.978474 (+ 3329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:04:38.983127 (+ 4653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:38.983139 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:38.983160 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:38.983860 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:38.984270 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:38.984275 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:38.984281 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:38.984534 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:38.985091 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:04:38.985100 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:04:38.985571 (+ 471us) client_negotiation.cc:769] Sending connection context
0812 01:04:38.985958 (+ 387us) client_negotiation.cc:240] Negotiation successful
0812 01:04:38.986543 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3385,"thread_start_us":3127,"threads_started":1}
I20260812 01:04:38.990446 29910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:04:38 dist-test-slave-7f01 krb5kdc[29651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496678, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.125@KRBTEST.COM for kudu/127.21.201.125@KRBTEST.COM
I20260812 01:04:39.008181 29916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:38.991510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:34241 (local address 127.0.0.1:44210)
0812 01:04:38.991727 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:38.991793 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 01:04:38.991914 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:04:38.992947 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:04:38.992955 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:04:38.993489 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:04:38.993702 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:38.993717 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:38.995903 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:38.995911 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:38.997402 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:38.997420 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:38.998606 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:38.998614 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:38.998921 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:38.999633 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:04:38.999661 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:04:39.002259 (+ 2598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:04:39.005596 (+ 3337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:39.005604 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:39.005610 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:39.005997 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:39.006372 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:39.006377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:39.006381 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:39.006517 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:39.007121 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:04:39.007126 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:04:39.007262 (+ 136us) client_negotiation.cc:769] Sending connection context
0812 01:04:39.007421 (+ 159us) client_negotiation.cc:240] Negotiation successful
0812 01:04:39.007711 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":27}
I20260812 01:04:39.008219 29917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:38.991602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44210 (local address 127.21.201.125:34241)
0812 01:04:38.992427 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:38.992436 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:38.992474 (+ 38us) server_negotiation.cc:407] Connection header received
0812 01:04:38.992533 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:38.992542 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:38.992774 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:38.992914 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:38.993916 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:38.995701 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:38.997571 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:38.998432 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:38.998614 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:39.002482 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:39.002493 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:39.002505 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:39.002533 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:39.005450 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:39.006130 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:39.006138 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:39.006141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:39.006257 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:39.006619 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:39.006626 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:39.006629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:39.006996 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:39.007192 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:39.007572 (+ 380us) server_negotiation.cc:299] Negotiation successful
0812 01:04:39.007843 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":648,"thread_start_us":493,"threads_started":1}
I20260812 01:04:39.010813 29910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } has no permanent_uuid. Determining permanent_uuid...
I20260812 01:04:39.011962 29916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.011453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:38401 (local address 127.0.0.1:60698)
0812 01:04:39.011637 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:39.011784 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 01:04:39.012248 29829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
I20260812 01:04:39.013819 29916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.013501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:38401 (local address 127.0.0.1:60704)
0812 01:04:39.013645 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:39.013724 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260812 01:04:39.014125 29910 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:38401: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.201.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.124.keytab.
Entry for principal kudu/127.21.201.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.124.keytab.
I20260812 01:04:39.032794 22309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
/tmp/dist-test-taskJMOGj2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.124:38401
--webserver_interface=127.21.201.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.124.keytab
--principal=kudu/127.21.201.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.201.84:43853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.201.126:42797,127.21.201.125:34241,127.21.201.124:38401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 01:04:39.063457 29910 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } attempt: 1
I20260812 01:04:39.064654 29916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.064192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:38401 (local address 127.0.0.1:60706)
0812 01:04:39.064389 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:39.064502 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260812 01:04:39.065737 29916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.065397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:38401 (local address 127.0.0.1:60716)
0812 01:04:39.065550 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:39.065636 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 01:04:39.066042 29910 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:38401: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
I20260812 01:04:39.088126 29785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 } attempt: 5
Aug 12 01:04:39 dist-test-slave-7f01 krb5kdc[29651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496678, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for kudu/127.21.201.125@KRBTEST.COM
I20260812 01:04:39.101933 29917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.088957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44226 (local address 127.21.201.125:34241)
0812 01:04:39.089146 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:39.089153 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:39.089262 (+ 109us) server_negotiation.cc:407] Connection header received
0812 01:04:39.089486 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:39.089496 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:39.089560 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:39.089672 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:39.090509 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.091225 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.092516 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.092957 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.093102 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:39.096355 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:39.096360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:39.096364 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:39.096395 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:39.099212 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:39.099892 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:39.099900 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:39.099905 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:39.100054 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:39.100467 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:39.100472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:39.100476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:39.100759 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:39.101025 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:39.101359 (+ 334us) server_negotiation.cc:299] Negotiation successful
0812 01:04:39.101586 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 01:04:39.101948 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.088870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:34241 (local address 127.0.0.1:44226)
0812 01:04:39.089154 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:39.089174 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:04:39.089332 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:04:39.089716 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:04:39.089722 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:04:39.090147 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:04:39.090339 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:39.090348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.091364 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.091371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:39.092362 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:39.092376 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.093107 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.093113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:39.093344 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:39.094027 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:04:39.094045 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:04:39.096161 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:04:39.099358 (+ 3197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:39.099364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:39.099368 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:39.099728 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:39.100159 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:39.100182 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:39.100186 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:39.100339 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:39.100881 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:04:39.100886 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:04:39.101068 (+ 182us) client_negotiation.cc:769] Sending connection context
0812 01:04:39.101234 (+ 166us) client_negotiation.cc:240] Negotiation successful
0812 01:04:39.101551 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260812 01:04:39.103327 29785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } has no permanent_uuid. Determining permanent_uuid...
I20260812 01:04:39.104391 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.104008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:38401 (local address 127.0.0.1:60720)
0812 01:04:39.104177 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:39.104263 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 01:04:39.105298 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.104966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:38401 (local address 127.0.0.1:60728)
0812 01:04:39.105122 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:39.105195 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 01:04:39.105536 29785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:38401: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
I20260812 01:04:39.134352 29910 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } attempt: 2
I20260812 01:04:39.135579 29916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.135096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:38401 (local address 127.0.0.1:60730)
0812 01:04:39.135317 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:39.135414 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260812 01:04:39.136730 29916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.136344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:38401 (local address 127.0.0.1:60738)
0812 01:04:39.136494 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:39.136573 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 01:04:39.137032 29910 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:38401: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
I20260812 01:04:39.156778 29785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } attempt: 1
I20260812 01:04:39.157931 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.157457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:38401 (local address 127.0.0.1:60750)
0812 01:04:39.157663 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:39.157761 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260812 01:04:39.158948 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.158631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:38401 (local address 127.0.0.1:60760)
0812 01:04:39.158779 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:39.158851 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 01:04:39.159194 29785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:38401: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
I20260812 01:04:39.227435 29785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } attempt: 2
I20260812 01:04:39.228274 29910 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } attempt: 3
I20260812 01:04:39.229023 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.228266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:38401 (local address 127.0.0.1:60770)
0812 01:04:39.228695 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:39.228812 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268}
I20260812 01:04:39.229338 29916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.228830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:38401 (local address 127.0.0.1:60784)
0812 01:04:39.229072 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:39.229166 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260812 01:04:39.230378 29916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.229931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:38401 (local address 127.0.0.1:60798)
0812 01:04:39.230181 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:39.230257 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
W20260812 01:04:39.230597 29910 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:38401: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
I20260812 01:04:39.231591 29924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.229946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:38401 (local address 127.0.0.1:60802)
0812 01:04:39.231146 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:39.231320 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1078,"thread_start_us":875,"threads_started":1}
W20260812 01:04:39.231894 29785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:38401: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
I20260812 01:04:39.338167 29785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } attempt: 3
I20260812 01:04:39.339646 29924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.339026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:38401 (local address 127.0.0.1:60812)
0812 01:04:39.339289 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:39.339417 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260812 01:04:39.341096 29924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.340544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:38401 (local address 127.0.0.1:60828)
0812 01:04:39.340785 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:39.340924 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260812 01:04:39.341372 29785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:38401: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
I20260812 01:04:39.373819 29910 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } attempt: 4
I20260812 01:04:39.375294 29916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.374615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:38401 (local address 127.0.0.1:60842)
0812 01:04:39.374947 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:39.375080 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260812 01:04:39.376505 29916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.376170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:38401 (local address 127.0.0.1:60856)
0812 01:04:39.376319 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:39.376402 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 01:04:39.376796 29910 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:38401: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
W20260812 01:04:39.432153 29922 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 01:04:39.432524 29922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 01:04:39.432657 29922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 01:04:39.432789 29922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 01:04:39.440493 29922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 01:04:39.440589 29922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 01:04:39.440685 29922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 01:04:39.440776 29922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 01:04:39.440891 29922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 01:04:39.451081 29922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.201.84:43853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.201.126:42797,127.21.201.125:34241,127.21.201.124:38401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/krb5kdc/kudu_127.21.201.124.keytab
--principal=kudu/127.21.201.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.201.124:38401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.201.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 00:04:17 UTC on ccycloud.quasar-okduwe.root.comops.site
build id 4581749
ASAN enabled
I20260812 01:04:39.452461 29922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 01:04:39.453823 29922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 01:04:39.468981 29934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:39.469485 29935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 01:04:39.470060 29941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 01:04:39.470297 29922 server_base.cc:1034] running on GCE node
I20260812 01:04:39.471168 29922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 01:04:39.472644 29922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 01:04:39.473915 29922 hybrid_clock.cc:648] HybridClock initialized: now 1786496679473837 us; error 51 us; skew 500 ppm
I20260812 01:04:39.474253 29922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 01:04:39 dist-test-slave-7f01 krb5kdc[29651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496679, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 01:04:39.482460 29922 init.cc:377] Logged in from keytab as kudu/127.21.201.124@KRBTEST.COM (short username kudu)
I20260812 01:04:39.484330 29922 webserver.cc:460] Webserver started at http://127.21.201.124:38647/ using document root <none> and password file <none>
I20260812 01:04:39.485102 29922 fs_manager.cc:359] Metadata directory not provided
I20260812 01:04:39.485241 29922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 01:04:39.485602 29922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 01:04:39.489138 29922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-2/data/instance:
uuid: "d64df0092ede4fc78390927cad08e1d2"
format_stamp: "Formatted at 2026-08-12 01:04:39 on dist-test-slave-7f01"
server_key: "1418cf5e191249cfcf1bfa46fddb44ea"
server_key_iv: "d2eb605a904f1e24f85d9896e3a0faa3"
server_key_version: "encryptionkey@0"
I20260812 01:04:39.490000 29922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-2/wal/instance:
uuid: "d64df0092ede4fc78390927cad08e1d2"
format_stamp: "Formatted at 2026-08-12 01:04:39 on dist-test-slave-7f01"
server_key: "1418cf5e191249cfcf1bfa46fddb44ea"
server_key_iv: "d2eb605a904f1e24f85d9896e3a0faa3"
server_key_version: "encryptionkey@0"
I20260812 01:04:39.495412 29922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 01:04:39.499030 29950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 01:04:39.500654 29922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 01:04:39.500837 29922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-2/data,/tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-2/wal
uuid: "d64df0092ede4fc78390927cad08e1d2"
format_stamp: "Formatted at 2026-08-12 01:04:39 on dist-test-slave-7f01"
server_key: "1418cf5e191249cfcf1bfa46fddb44ea"
server_key_iv: "d2eb605a904f1e24f85d9896e3a0faa3"
server_key_version: "encryptionkey@0"
I20260812 01:04:39.501070 29922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 01:04:39.518616 29785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } attempt: 4
I20260812 01:04:39.519958 29924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.519432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:38401 (local address 127.0.0.1:60868)
0812 01:04:39.519662 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:39.519786 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260812 01:04:39.521082 29924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.520690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:38401 (local address 127.0.0.1:60872)
0812 01:04:39.520860 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:39.520956 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 01:04:39.521319 29785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.201.124:38401: Network error: Client connection negotiation failed: client connection to 127.21.201.124:38401: connect: Connection refused (error 111)
I20260812 01:04:39.526633 29922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 01:04:39.531093 29922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 01:04:39.531374 29922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 01:04:39.552066 29922 rpc_server.cc:307] RPC server started. Bound to: 127.21.201.124:38401
I20260812 01:04:39.552099 30038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.201.124:38401 every 8 connection(s)
I20260812 01:04:39.553400 29922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-2/data/info.pb
I20260812 01:04:39.557351 30039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 01:04:39.561648 22309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu as pid 29922
I20260812 01:04:39.561872 22309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJMOGj2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786496278741276-22309-0/minicluster-data/master-2/wal/instance
I20260812 01:04:39.562462 22309 external_mini_cluster.cc:1428] Setting key 3e32e574333863e5e531d06cd7f16ec0
I20260812 01:04:39.565425 30039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42797 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:04:39 dist-test-slave-7f01 krb5kdc[29651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496679, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.124@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
Aug 12 01:04:39 dist-test-slave-7f01 krb5kdc[29651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.124@KRBTEST.COM
I20260812 01:04:39.602222 30044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.570248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42860 (local address 127.21.201.126:42797)
0812 01:04:39.579970 (+ 9722us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:39.579980 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:39.582973 (+ 2993us) server_negotiation.cc:407] Connection header received
0812 01:04:39.583057 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:39.583061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:39.583129 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:39.583264 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:39.585696 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.586539 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.589003 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.589547 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.589694 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:39.594430 (+ 4736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:39.594436 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:39.594439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:39.594474 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:39.598187 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:39.599164 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:39.599252 (+ 88us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:39.599255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:39.599445 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:39.599998 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:39.600003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:39.600006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:39.600453 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:39.600607 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:39.601565 (+ 958us) server_negotiation.cc:299] Negotiation successful
0812 01:04:39.601804 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9549,"thread_start_us":9314,"threads_started":1}
I20260812 01:04:39.603824 30043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.569595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.126:42797 (local address 127.0.0.1:42860)
0812 01:04:39.580640 (+ 11045us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:39.580687 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 01:04:39.582688 (+ 2001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:04:39.583573 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:04:39.583590 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:04:39.584562 (+ 972us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:04:39.585502 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:39.585522 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.586750 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.586756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:39.588810 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:39.588825 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.589802 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.589810 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:39.590034 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:39.590939 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:04:39.590983 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:04:39.594186 (+ 3203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:04:39.598333 (+ 4147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:39.598347 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:39.598361 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:39.599005 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:39.599571 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:39.599576 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:39.599580 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:39.599841 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:39.600602 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:04:39.600610 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:04:39.601088 (+ 478us) client_negotiation.cc:769] Sending connection context
0812 01:04:39.601423 (+ 335us) client_negotiation.cc:240] Negotiation successful
0812 01:04:39.601955 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10721,"mutex_wait_us":272,"thread_start_us":10565,"threads_started":1}
I20260812 01:04:39.604413 30042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.564778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60884 (local address 127.21.201.124:38401)
0812 01:04:39.583936 (+ 19158us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:39.583943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:39.583978 (+ 35us) server_negotiation.cc:407] Connection header received
0812 01:04:39.584041 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:39.584052 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:39.584292 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:39.584509 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:39.586423 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.587874 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.589189 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.590026 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.590150 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:39.595275 (+ 5125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:39.595289 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:39.595308 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:39.595357 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:39.600056 (+ 4699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:39.601024 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:39.601032 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:39.601035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:39.601196 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:39.601646 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:39.601651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:39.601653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:39.602058 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:39.602380 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:39.603574 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0812 01:04:39.603906 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":18909,"spinlock_wait_cycles":16640,"thread_start_us":18672,"threads_started":1}
I20260812 01:04:39.605667 30039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 01:04:39 dist-test-slave-7f01 krb5kdc[29651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496679, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.124@KRBTEST.COM for kudu/127.21.201.125@KRBTEST.COM
Aug 12 01:04:39 dist-test-slave-7f01 krb5kdc[29651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.126@KRBTEST.COM
I20260812 01:04:39.622794 30043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.606629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.125:34241 (local address 127.0.0.1:44230)
0812 01:04:39.606805 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:39.606826 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 01:04:39.606928 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:04:39.608902 (+ 1974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:04:39.608908 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:04:39.609386 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:04:39.609608 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:39.609623 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.611055 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.611061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:39.612366 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:39.612382 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.613264 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.613272 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:39.613500 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:39.614087 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:04:39.614104 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:04:39.616213 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:04:39.619986 (+ 3773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:39.619991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:39.619995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:39.620479 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:39.620929 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:39.620934 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:39.620938 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:39.621151 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:39.621810 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:04:39.621815 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:04:39.622009 (+ 194us) client_negotiation.cc:769] Sending connection context
0812 01:04:39.622172 (+ 163us) client_negotiation.cc:240] Negotiation successful
0812 01:04:39.622445 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260812 01:04:39.623464 30045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.607079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44230 (local address 127.21.201.125:34241)
0812 01:04:39.608564 (+ 1485us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:39.608570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:39.608589 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:04:39.608650 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:39.608654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:39.608732 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:39.608854 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:39.609832 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.610835 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.612535 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.613110 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.613589 (+ 479us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:39.616418 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:39.616424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:39.616428 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:39.616460 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:39.619833 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:39.620625 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:39.620632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:39.620635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:39.620799 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:39.621285 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:39.621292 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:39.621299 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:39.621667 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:39.621824 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:39.622289 (+ 465us) server_negotiation.cc:299] Negotiation successful
0812 01:04:39.622517 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1358,"thread_start_us":1229,"threads_started":1}
I20260812 01:04:39.624598 30039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } has no permanent_uuid. Determining permanent_uuid...
I20260812 01:04:39.625089 30044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.608030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42876 (local address 127.21.201.126:42797)
0812 01:04:39.608249 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:39.608257 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:39.608281 (+ 24us) server_negotiation.cc:407] Connection header received
0812 01:04:39.608352 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:39.608357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:39.608454 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:39.608579 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:39.610124 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.610824 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.612126 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.612643 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.612758 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:39.618278 (+ 5520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:39.618286 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:39.618291 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:39.618331 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:39.621934 (+ 3603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:39.622764 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:39.622770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:39.622774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:39.622936 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:39.623417 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:39.623423 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:39.623426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:39.623823 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:39.624131 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:39.624467 (+ 336us) server_negotiation.cc:299] Negotiation successful
0812 01:04:39.624724 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Aug 12 01:04:39 dist-test-slave-7f01 krb5kdc[29651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496679, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.124@KRBTEST.COM for kudu/127.21.201.124@KRBTEST.COM
Aug 12 01:04:39 dist-test-slave-7f01 krb5kdc[29651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.201.125@KRBTEST.COM
I20260812 01:04:39.640609 30043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.625337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:38401 (local address 127.0.0.1:60888)
0812 01:04:39.625578 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:39.625596 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 01:04:39.625697 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:04:39.626169 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:04:39.626174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:04:39.626649 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:04:39.626857 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:39.626871 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.628267 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.628273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:39.629514 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:39.629526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.630463 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.630469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:39.630688 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:39.631376 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:04:39.631398 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:04:39.634194 (+ 2796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:04:39.637862 (+ 3668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:39.637867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:39.637871 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:39.638440 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:39.638885 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:39.638889 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:39.638892 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:39.639050 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:39.639685 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:04:39.639689 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:04:39.639866 (+ 177us) client_negotiation.cc:769] Sending connection context
0812 01:04:39.640011 (+ 145us) client_negotiation.cc:240] Negotiation successful
0812 01:04:39.640259 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260812 01:04:39.641469 30042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.625427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60888 (local address 127.21.201.124:38401)
0812 01:04:39.625623 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:39.625628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:39.625647 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:04:39.625821 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:39.625826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:39.625896 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:39.626005 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:39.627034 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.628118 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.629659 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.630235 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.630473 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:39.634423 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:39.634431 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:39.634438 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:39.634481 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:39.637704 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:39.638572 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:39.638577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:39.638580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:39.638767 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:39.639164 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:39.639168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:39.639171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:39.639492 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:39.639638 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:39.640922 (+ 1284us) server_negotiation.cc:299] Negotiation successful
0812 01:04:39.641183 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 01:04:39.644129 30045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.627108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44234 (local address 127.21.201.125:34241)
0812 01:04:39.627305 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:39.627312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:39.627333 (+ 21us) server_negotiation.cc:407] Connection header received
0812 01:04:39.627528 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:39.627534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:39.627606 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:39.627705 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:39.629184 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.630202 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.631860 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.632265 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.632424 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:39.637593 (+ 5169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:39.637600 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:39.637609 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:39.637654 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:39.640593 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:39.641866 (+ 1273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:39.641872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:39.641875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:39.642065 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:39.642539 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:39.642544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:39.642547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:39.642897 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:39.643178 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:39.643527 (+ 349us) server_negotiation.cc:299] Negotiation successful
0812 01:04:39.643790 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260812 01:04:39.646749 30039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d64df0092ede4fc78390927cad08e1d2: Bootstrap starting.
I20260812 01:04:39.650005 30039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d64df0092ede4fc78390927cad08e1d2: Neither blocks nor log segments found. Creating new log.
I20260812 01:04:39.651364 30039 log.cc:824] T 00000000000000000000000000000000 P d64df0092ede4fc78390927cad08e1d2: Log is configured to *not* fsync() on all Append() calls
I20260812 01:04:39.655105 22309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 01:04:39.655217 30039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d64df0092ede4fc78390927cad08e1d2: No bootstrap required, opened a new log
I20260812 01:04:39.658610 29745 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "98710605273c460790bc328105af2091"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42876
I20260812 01:04:39.665117 30039 raft_consensus.cc:348] T 00000000000000000000000000000000 P d64df0092ede4fc78390927cad08e1d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98710605273c460790bc328105af2091" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42797 } } peers { permanent_uuid: "2299e87e6e384819a6ef59d216aaa232" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 } } peers { permanent_uuid: "d64df0092ede4fc78390927cad08e1d2" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } }
I20260812 01:04:39.665583 30039 raft_consensus.cc:729] T 00000000000000000000000000000000 P d64df0092ede4fc78390927cad08e1d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d64df0092ede4fc78390927cad08e1d2, State: Initialized, Role: FOLLOWER
I20260812 01:04:39.666040 30039 consensus_queue.cc:260] T 00000000000000000000000000000000 P d64df0092ede4fc78390927cad08e1d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98710605273c460790bc328105af2091" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42797 } } peers { permanent_uuid: "2299e87e6e384819a6ef59d216aaa232" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 } } peers { permanent_uuid: "d64df0092ede4fc78390927cad08e1d2" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } }
I20260812 01:04:39.667028 30048 sys_catalog.cc:455] T 00000000000000000000000000000000 P d64df0092ede4fc78390927cad08e1d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98710605273c460790bc328105af2091" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42797 } } peers { permanent_uuid: "2299e87e6e384819a6ef59d216aaa232" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 } } peers { permanent_uuid: "d64df0092ede4fc78390927cad08e1d2" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } } }
I20260812 01:04:39.667387 30048 sys_catalog.cc:458] T 00000000000000000000000000000000 P d64df0092ede4fc78390927cad08e1d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:04:39.668372 30039 sys_catalog.cc:564] T 00000000000000000000000000000000 P d64df0092ede4fc78390927cad08e1d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:04:39.676055 29910 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } attempt: 5
W20260812 01:04:39.681658 30063 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d64df0092ede4fc78390927cad08e1d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 01:04:39.681797 30063 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 12 01:04:39 dist-test-slave-7f01 krb5kdc[29651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496678, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.125@KRBTEST.COM for kudu/127.21.201.124@KRBTEST.COM
I20260812 01:04:39.680999 29922 master_runner.cc:186] Error getting master registration for 127.21.201.126:42797: OK, instance_id { permanent_uuid: "98710605273c460790bc328105af2091" instance_seqno: 1786496678376392 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 01:04:39.693326 29922 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 01:04:39.697517 29916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.676737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:38401 (local address 127.0.0.1:60890)
0812 01:04:39.677033 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:39.677053 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 01:04:39.677185 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:04:39.677571 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:04:39.677576 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:04:39.678163 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:04:39.678421 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:39.678436 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.680066 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.680072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:39.681824 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:39.681840 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.683729 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.683778 (+ 49us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:39.684331 (+ 553us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:39.685338 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:04:39.685369 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:04:39.689868 (+ 4499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:04:39.694334 (+ 4466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:39.694346 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:39.694350 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:39.694884 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:39.695343 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:39.695348 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:39.695351 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:39.695567 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:39.696376 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:04:39.696382 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:04:39.696567 (+ 185us) client_negotiation.cc:769] Sending connection context
0812 01:04:39.696748 (+ 181us) client_negotiation.cc:240] Negotiation successful
0812 01:04:39.697111 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260812 01:04:39.697551 30042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.676916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60890 (local address 127.21.201.124:38401)
0812 01:04:39.677085 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:39.677093 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:39.677116 (+ 23us) server_negotiation.cc:407] Connection header received
0812 01:04:39.677317 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:39.677323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:39.677416 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:39.677538 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:39.678624 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.679861 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.682004 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.682961 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.683192 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:39.690084 (+ 6892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:39.690089 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:39.690093 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:39.690125 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:39.694166 (+ 4041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:39.695015 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:39.695020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:39.695024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:39.695205 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:39.695689 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:39.695694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:39.695697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:39.696230 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:39.696443 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:39.696865 (+ 422us) server_negotiation.cc:299] Negotiation successful
0812 01:04:39.697142 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 01:04:39.704116 29910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2299e87e6e384819a6ef59d216aaa232: Bootstrap starting.
I20260812 01:04:39.708294 29910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2299e87e6e384819a6ef59d216aaa232: Neither blocks nor log segments found. Creating new log.
I20260812 01:04:39.709578 29910 log.cc:824] T 00000000000000000000000000000000 P 2299e87e6e384819a6ef59d216aaa232: Log is configured to *not* fsync() on all Append() calls
I20260812 01:04:39.713340 29910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2299e87e6e384819a6ef59d216aaa232: No bootstrap required, opened a new log
I20260812 01:04:39.724519 29910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2299e87e6e384819a6ef59d216aaa232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98710605273c460790bc328105af2091" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42797 } } peers { permanent_uuid: "2299e87e6e384819a6ef59d216aaa232" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 } } peers { permanent_uuid: "d64df0092ede4fc78390927cad08e1d2" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } }
I20260812 01:04:39.724967 29910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2299e87e6e384819a6ef59d216aaa232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2299e87e6e384819a6ef59d216aaa232, State: Initialized, Role: FOLLOWER
I20260812 01:04:39.725541 29910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2299e87e6e384819a6ef59d216aaa232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98710605273c460790bc328105af2091" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42797 } } peers { permanent_uuid: "2299e87e6e384819a6ef59d216aaa232" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 } } peers { permanent_uuid: "d64df0092ede4fc78390927cad08e1d2" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } }
I20260812 01:04:39.726832 30067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2299e87e6e384819a6ef59d216aaa232 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98710605273c460790bc328105af2091" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42797 } } peers { permanent_uuid: "2299e87e6e384819a6ef59d216aaa232" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 } } peers { permanent_uuid: "d64df0092ede4fc78390927cad08e1d2" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } } }
I20260812 01:04:39.727281 30067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2299e87e6e384819a6ef59d216aaa232 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:04:39.728040 29910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2299e87e6e384819a6ef59d216aaa232 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:04:39.736093 29794 master_runner.cc:186] Error getting master registration for 127.21.201.126:42797: OK, instance_id { permanent_uuid: "98710605273c460790bc328105af2091" instance_seqno: 1786496678376392 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 01:04:39.747128 29794 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 01:04:39.749841 30087 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2299e87e6e384819a6ef59d216aaa232: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 01:04:39.749974 30087 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260812 01:04:39.783681 29958 tablet.cc:2369] T 00000000000000000000000000000000 P d64df0092ede4fc78390927cad08e1d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:04:39.798540 29785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } attempt: 5
Aug 12 01:04:39 dist-test-slave-7f01 krb5kdc[29651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786496678, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.201.126@KRBTEST.COM for kudu/127.21.201.124@KRBTEST.COM
I20260812 01:04:39.816179 29924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.799381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.201.124:38401 (local address 127.0.0.1:60906)
0812 01:04:39.799640 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0812 01:04:39.799664 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 01:04:39.799959 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 01:04:39.800665 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 01:04:39.800673 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 01:04:39.801249 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0812 01:04:39.801555 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:39.801567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.802669 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.802677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:39.804182 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 01:04:39.804209 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.805315 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.805320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 01:04:39.805548 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:39.806234 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 01:04:39.806257 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 01:04:39.809115 (+ 2858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 01:04:39.812509 (+ 3394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:39.812516 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:39.812522 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:39.813136 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:39.813620 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 01:04:39.813630 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 01:04:39.813633 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 01:04:39.813797 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 01:04:39.814878 (+ 1081us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 01:04:39.814884 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 01:04:39.815096 (+ 212us) client_negotiation.cc:769] Sending connection context
0812 01:04:39.815305 (+ 209us) client_negotiation.cc:240] Negotiation successful
0812 01:04:39.815656 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260812 01:04:39.816215 30042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.800133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60906 (local address 127.21.201.124:38401)
0812 01:04:39.800334 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:39.800341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:39.800360 (+ 19us) server_negotiation.cc:407] Connection header received
0812 01:04:39.800427 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:39.800431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:39.800493 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:39.800617 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:39.801726 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.802520 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.804407 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.805166 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.805299 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:39.809318 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:39.809323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:39.809326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:39.809362 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:39.812340 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:39.813298 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:39.813303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:39.813309 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:39.813501 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:39.814007 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:39.814012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:39.814015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:39.814555 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:39.814746 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:39.815467 (+ 721us) server_negotiation.cc:299] Negotiation successful
0812 01:04:39.815788 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 01:04:39.823478 29785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091: Bootstrap starting.
I20260812 01:04:39.828387 29785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091: Neither blocks nor log segments found. Creating new log.
I20260812 01:04:39.830014 29785 log.cc:824] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091: Log is configured to *not* fsync() on all Append() calls
I20260812 01:04:39.840188 29785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091: No bootstrap required, opened a new log
I20260812 01:04:39.849195 29785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98710605273c460790bc328105af2091" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42797 } } peers { permanent_uuid: "2299e87e6e384819a6ef59d216aaa232" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 } } peers { permanent_uuid: "d64df0092ede4fc78390927cad08e1d2" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } }
I20260812 01:04:39.849694 29785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98710605273c460790bc328105af2091, State: Initialized, Role: FOLLOWER
I20260812 01:04:39.850268 29785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98710605273c460790bc328105af2091" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42797 } } peers { permanent_uuid: "2299e87e6e384819a6ef59d216aaa232" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 } } peers { permanent_uuid: "d64df0092ede4fc78390927cad08e1d2" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } }
I20260812 01:04:39.851912 30090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98710605273c460790bc328105af2091" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42797 } } peers { permanent_uuid: "2299e87e6e384819a6ef59d216aaa232" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 } } peers { permanent_uuid: "d64df0092ede4fc78390927cad08e1d2" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } } }
I20260812 01:04:39.852821 30090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:04:39.853288 29785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 01:04:39.853762 29745 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 01:04:39.854007 29745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:04:39.858783 29745 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98710605273c460790bc328105af2091" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42797 } } peers { permanent_uuid: "2299e87e6e384819a6ef59d216aaa232" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 } } peers { permanent_uuid: "d64df0092ede4fc78390927cad08e1d2" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } }
I20260812 01:04:39.861882 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98710605273c460790bc328105af2091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2299e87e6e384819a6ef59d216aaa232"
I20260812 01:04:39.862594 29874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2299e87e6e384819a6ef59d216aaa232 [term 0 FOLLOWER]: Advancing to term 1
W20260812 01:04:39.865597 29706 tablet.cc:2369] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:04:39.872021 29874 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2299e87e6e384819a6ef59d216aaa232 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98710605273c460790bc328105af2091 in term 1.
I20260812 01:04:39.874109 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98710605273c460790bc328105af2091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d64df0092ede4fc78390927cad08e1d2"
I20260812 01:04:39.875023 30006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d64df0092ede4fc78390927cad08e1d2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 01:04:39.890138 30006 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d64df0092ede4fc78390927cad08e1d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98710605273c460790bc328105af2091 in term 1.
I20260812 01:04:39.891260 29701 leader_election.cc:302] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98710605273c460790bc328105af2091, d64df0092ede4fc78390927cad08e1d2; no voters:
I20260812 01:04:39.893014 29745 leader_election.cc:288] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091 [CANDIDATE]: Term 1 election: Requested vote from peers 2299e87e6e384819a6ef59d216aaa232 (127.21.201.125:34241), d64df0092ede4fc78390927cad08e1d2 (127.21.201.124:38401)
I20260812 01:04:39.909117 30090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 01:04:39.912708 30090 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091 [term 1 LEADER]: Becoming Leader. State: Replica: 98710605273c460790bc328105af2091, State: Running, Role: LEADER
I20260812 01:04:39.913805 30090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98710605273c460790bc328105af2091" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42797 } } peers { permanent_uuid: "2299e87e6e384819a6ef59d216aaa232" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 } } peers { permanent_uuid: "d64df0092ede4fc78390927cad08e1d2" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } }
W20260812 01:04:39.921657 29834 tablet.cc:2369] T 00000000000000000000000000000000 P 2299e87e6e384819a6ef59d216aaa232: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 01:04:39.933651 30100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98710605273c460790bc328105af2091. Latest consensus state: current_term: 1 leader_uuid: "98710605273c460790bc328105af2091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98710605273c460790bc328105af2091" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42797 } } peers { permanent_uuid: "2299e87e6e384819a6ef59d216aaa232" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 } } peers { permanent_uuid: "d64df0092ede4fc78390927cad08e1d2" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } } }
I20260812 01:04:39.934307 30100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091 [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:39.937516 30117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 01:04:39.964463 30044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.936369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42886 (local address 127.21.201.126:42797)
0812 01:04:39.936585 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:39.936594 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:39.937761 (+ 1167us) server_negotiation.cc:407] Connection header received
0812 01:04:39.938255 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:39.938288 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:39.938407 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:39.938503 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:39.941483 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.943041 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.945701 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.946588 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.946711 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:39.951196 (+ 4485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:39.951205 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:39.951213 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:39.951282 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:39.958506 (+ 7224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:39.960362 (+ 1856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:39.960372 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:39.960379 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:39.960624 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:39.961193 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:39.961206 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:39.961210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:39.961681 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:39.962019 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:39.963362 (+ 1343us) server_negotiation.cc:299] Negotiation successful
0812 01:04:39.963824 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 01:04:39.971475 30117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 01:04:40.002170 30045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.936385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44238 (local address 127.21.201.125:34241)
0812 01:04:39.936792 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:39.936801 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:39.946795 (+ 9994us) server_negotiation.cc:407] Connection header received
0812 01:04:39.952101 (+ 5306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:39.952111 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:39.952224 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:39.952341 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:39.957393 (+ 5052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.959091 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.970593 (+ 11502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.971361 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.971484 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:39.982655 (+ 11171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:39.982663 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:39.982669 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:39.982725 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:39.987061 (+ 4336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:39.995422 (+ 8361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:39.995429 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:39.995433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:39.995630 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:39.996654 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:39.996660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:39.996664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:39.997558 (+ 894us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:39.997885 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:40.001419 (+ 3534us) server_negotiation.cc:299] Negotiation successful
0812 01:04:40.001746 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260812 01:04:40.006947 30042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:39.947430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60920 (local address 127.21.201.124:38401)
0812 01:04:39.947664 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:39.947672 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:39.952592 (+ 4920us) server_negotiation.cc:407] Connection header received
0812 01:04:39.952753 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:39.952760 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:39.952871 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:39.952986 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:39.967093 (+ 14107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.968435 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.977326 (+ 8891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:39.978360 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:39.978515 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:39.988217 (+ 9702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 01:04:39.988227 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 01:04:39.988236 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 01:04:39.988317 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0812 01:04:39.994003 (+ 5686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:39.996228 (+ 2225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:39.996234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:39.996245 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:39.996465 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 01:04:39.999530 (+ 3065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 01:04:39.999538 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 01:04:39.999543 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 01:04:40.000197 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 01:04:40.000656 (+ 459us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:40.005899 (+ 5243us) server_negotiation.cc:299] Negotiation successful
0812 01:04:40.006310 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 01:04:40.015126 29874 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2299e87e6e384819a6ef59d216aaa232 [term 1 FOLLOWER]: Refusing update from remote peer 98710605273c460790bc328105af2091: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:04:40.020598 30006 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d64df0092ede4fc78390927cad08e1d2 [term 1 FOLLOWER]: Refusing update from remote peer 98710605273c460790bc328105af2091: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 01:04:40.023466 30100 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2299e87e6e384819a6ef59d216aaa232" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:04:40.024724 30100 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d64df0092ede4fc78390927cad08e1d2" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 01:04:40.062443 30067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2299e87e6e384819a6ef59d216aaa232 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98710605273c460790bc328105af2091. Latest consensus state: current_term: 1 leader_uuid: "98710605273c460790bc328105af2091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98710605273c460790bc328105af2091" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42797 } } peers { permanent_uuid: "2299e87e6e384819a6ef59d216aaa232" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 } } peers { permanent_uuid: "d64df0092ede4fc78390927cad08e1d2" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } } }
I20260812 01:04:40.062942 30067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2299e87e6e384819a6ef59d216aaa232 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:04:40.085588 30067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2299e87e6e384819a6ef59d216aaa232 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98710605273c460790bc328105af2091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98710605273c460790bc328105af2091" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42797 } } peers { permanent_uuid: "2299e87e6e384819a6ef59d216aaa232" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 } } peers { permanent_uuid: "d64df0092ede4fc78390927cad08e1d2" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } } }
I20260812 01:04:40.086006 30067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2299e87e6e384819a6ef59d216aaa232 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:04:40.105104 30090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98710605273c460790bc328105af2091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98710605273c460790bc328105af2091" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42797 } } peers { permanent_uuid: "2299e87e6e384819a6ef59d216aaa232" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 } } peers { permanent_uuid: "d64df0092ede4fc78390927cad08e1d2" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } } }
I20260812 01:04:40.105538 30090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091 [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:40.114512 30117 catalog_manager.cc:1347] Generated new cluster ID: f4775eb95a484c448fa04567481155bf
I20260812 01:04:40.114653 30117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 01:04:40.127686 30048 sys_catalog.cc:455] T 00000000000000000000000000000000 P d64df0092ede4fc78390927cad08e1d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98710605273c460790bc328105af2091. Latest consensus state: current_term: 1 leader_uuid: "98710605273c460790bc328105af2091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98710605273c460790bc328105af2091" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42797 } } peers { permanent_uuid: "2299e87e6e384819a6ef59d216aaa232" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 } } peers { permanent_uuid: "d64df0092ede4fc78390927cad08e1d2" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } } }
I20260812 01:04:40.128297 30048 sys_catalog.cc:458] T 00000000000000000000000000000000 P d64df0092ede4fc78390927cad08e1d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:04:40.129148 30100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98710605273c460790bc328105af2091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98710605273c460790bc328105af2091" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42797 } } peers { permanent_uuid: "2299e87e6e384819a6ef59d216aaa232" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 } } peers { permanent_uuid: "d64df0092ede4fc78390927cad08e1d2" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } } }
I20260812 01:04:40.129706 30100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091 [sys.catalog]: This master's current role is: LEADER
I20260812 01:04:40.145054 30048 sys_catalog.cc:455] T 00000000000000000000000000000000 P d64df0092ede4fc78390927cad08e1d2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98710605273c460790bc328105af2091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98710605273c460790bc328105af2091" member_type: VOTER last_known_addr { host: "127.21.201.126" port: 42797 } } peers { permanent_uuid: "2299e87e6e384819a6ef59d216aaa232" member_type: VOTER last_known_addr { host: "127.21.201.125" port: 34241 } } peers { permanent_uuid: "d64df0092ede4fc78390927cad08e1d2" member_type: VOTER last_known_addr { host: "127.21.201.124" port: 38401 } } }
I20260812 01:04:40.145388 30048 sys_catalog.cc:458] T 00000000000000000000000000000000 P d64df0092ede4fc78390927cad08e1d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 01:04:40.163076 30117 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 01:04:40.164839 30117 catalog_manager.cc:1504] Loading token signing keys...
I20260812 01:04:40.178076 30117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98710605273c460790bc328105af2091: Generated new TSK 0
I20260812 01:04:40.179040 30117 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 01:04:40.358343 22309 mini_kdc.cc:348] Time spent kdestroy: real 0.151s user 0.001s sys 0.029s
W20260812 01:04:40.411891 30045 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:04:40.401601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44242 (local address 127.21.201.125:34241)
0812 01:04:40.401829 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:40.401837 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:40.408715 (+ 6878us) server_negotiation.cc:407] Connection header received
0812 01:04:40.410112 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:40.410119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:40.410200 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:40.410285 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:40.411580 (+ 1295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44242: BlockingRecv error: recv got EOF from 127.0.0.1:44242 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260812 01:04:40.412276 30044 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:04:40.398091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42898 (local address 127.21.201.126:42797)
0812 01:04:40.398378 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:40.398386 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:40.406312 (+ 7926us) server_negotiation.cc:407] Connection header received
0812 01:04:40.406533 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:40.406542 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:40.406653 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:40.406817 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:40.411961 (+ 5144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42898: BlockingRecv error: recv got EOF from 127.0.0.1:42898 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260812 01:04:40.415544 30042 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:04:40.409292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60928 (local address 127.21.201.124:38401)
0812 01:04:40.409504 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:40.409510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:40.413357 (+ 3847us) server_negotiation.cc:407] Connection header received
0812 01:04:40.414172 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:40.414179 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:40.414267 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:40.414337 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:40.415268 (+ 931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60928: BlockingRecv error: recv got EOF from 127.0.0.1:60928 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260812 01:04:40.459769 30045 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:04:40.450402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44258 (local address 127.21.201.125:34241)
0812 01:04:40.450669 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:40.450677 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:40.457709 (+ 7032us) server_negotiation.cc:407] Connection header received
0812 01:04:40.458209 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:40.458214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:40.458343 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:40.458431 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:40.459511 (+ 1080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44258: BlockingRecv error: recv got EOF from 127.0.0.1:44258 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260812 01:04:40.473423 30045 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:04:40.471175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44272 (local address 127.21.201.125:34241)
0812 01:04:40.471401 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:40.471409 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:40.471915 (+ 506us) server_negotiation.cc:407] Connection header received
0812 01:04:40.472131 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:40.472138 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:40.472225 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:40.472340 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:40.473172 (+ 832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44272: BlockingRecv error: recv got EOF from 127.0.0.1:44272 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260812 01:04:40.481164 30045 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:04:40.478236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44276 (local address 127.21.201.125:34241)
0812 01:04:40.478455 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:40.478465 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:40.479399 (+ 934us) server_negotiation.cc:407] Connection header received
0812 01:04:40.479724 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:40.479730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:40.479849 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:40.479924 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:40.480893 (+ 969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44276: BlockingRecv error: recv got EOF from 127.0.0.1:44276 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260812 01:04:40.491776 30045 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:04:40.489546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44290 (local address 127.21.201.125:34241)
0812 01:04:40.489794 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:40.489803 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:40.490363 (+ 560us) server_negotiation.cc:407] Connection header received
0812 01:04:40.490599 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:40.490604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:40.490673 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:40.490788 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:40.491516 (+ 728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44290: BlockingRecv error: recv got EOF from 127.0.0.1:44290 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260812 01:04:40.505654 30045 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:04:40.502941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44294 (local address 127.21.201.125:34241)
0812 01:04:40.503190 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:40.503199 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:40.503774 (+ 575us) server_negotiation.cc:407] Connection header received
0812 01:04:40.504198 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:40.504204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:40.504336 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:40.504414 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:40.505369 (+ 955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44294: BlockingRecv error: recv got EOF from 127.0.0.1:44294 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260812 01:04:40.526968 30045 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:04:40.524806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44296 (local address 127.21.201.125:34241)
0812 01:04:40.524999 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:40.525008 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:40.525564 (+ 556us) server_negotiation.cc:407] Connection header received
0812 01:04:40.525778 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:40.525784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:40.525868 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:40.525990 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:40.526706 (+ 716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44296: BlockingRecv error: recv got EOF from 127.0.0.1:44296 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260812 01:04:40.594262 30045 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:04:40.591923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44312 (local address 127.21.201.125:34241)
0812 01:04:40.592174 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:40.592184 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:40.592777 (+ 593us) server_negotiation.cc:407] Connection header received
0812 01:04:40.593021 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:40.593026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:40.593106 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:40.593223 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:40.594024 (+ 801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44312: BlockingRecv error: recv got EOF from 127.0.0.1:44312 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260812 01:04:40.671857 30045 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 01:04:40.669216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44324 (local address 127.21.201.125:34241)
0812 01:04:40.669427 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:40.669434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:40.670047 (+ 613us) server_negotiation.cc:407] Connection header received
0812 01:04:40.670273 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:40.670282 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:40.670424 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:40.670560 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 01:04:40.671498 (+ 938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44324: BlockingRecv error: recv got EOF from 127.0.0.1:44324 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 01:04:40.684650 30063 catalog_manager.cc:1259] Loaded cluster ID: f4775eb95a484c448fa04567481155bf
I20260812 01:04:40.684787 30063 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d64df0092ede4fc78390927cad08e1d2: loading cluster ID for follower catalog manager: success
I20260812 01:04:40.689234 30063 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d64df0092ede4fc78390927cad08e1d2: acquiring CA information for follower catalog manager: success
I20260812 01:04:40.692097 30063 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d64df0092ede4fc78390927cad08e1d2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 01:04:40.752097 30087 catalog_manager.cc:1259] Loaded cluster ID: f4775eb95a484c448fa04567481155bf
I20260812 01:04:40.752231 30087 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2299e87e6e384819a6ef59d216aaa232: loading cluster ID for follower catalog manager: success
I20260812 01:04:40.755064 30087 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2299e87e6e384819a6ef59d216aaa232: acquiring CA information for follower catalog manager: success
I20260812 01:04:40.756901 30087 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2299e87e6e384819a6ef59d216aaa232: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 01:04:40.809698 30045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:40.803799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44338 (local address 127.21.201.125:34241)
0812 01:04:40.804056 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:40.804063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:40.804362 (+ 299us) server_negotiation.cc:407] Connection header received
0812 01:04:40.804539 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:40.804543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:40.804884 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:40.804990 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:04:40.805390 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:40.806218 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:40.807585 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:40.807993 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:40.808091 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:40.808569 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:04:40.809077 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:04:40.809184 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:40.809284 (+ 100us) server_negotiation.cc:299] Negotiation successful
0812 01:04:40.809355 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 01:04:40.821017 30042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:40.815545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60944 (local address 127.21.201.124:38401)
0812 01:04:40.815819 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:40.815824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:40.816026 (+ 202us) server_negotiation.cc:407] Connection header received
0812 01:04:40.816204 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:40.816208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:40.816365 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:40.816493 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:04:40.816847 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:40.817537 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:40.818851 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:40.819247 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:40.819335 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:40.819859 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:04:40.820369 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:04:40.820484 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:40.820586 (+ 102us) server_negotiation.cc:299] Negotiation successful
0812 01:04:40.820660 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 01:04:40.834338 30045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:40.827129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44340 (local address 127.21.201.125:34241)
0812 01:04:40.827368 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:40.827377 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:40.828064 (+ 687us) server_negotiation.cc:407] Connection header received
0812 01:04:40.828326 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:40.828335 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:40.828445 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:40.828510 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:04:40.829366 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:40.830397 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:40.832416 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:40.832889 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:40.833015 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:40.833486 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:04:40.833647 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:04:40.833745 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:40.833956 (+ 211us) server_negotiation.cc:299] Negotiation successful
0812 01:04:40.834042 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 01:04:40.835388 30042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:40.828233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60952 (local address 127.21.201.124:38401)
0812 01:04:40.828751 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:40.828757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:40.828862 (+ 105us) server_negotiation.cc:407] Connection header received
0812 01:04:40.829108 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:40.829113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:40.829186 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:40.829282 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:04:40.829934 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:40.830926 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:40.833191 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:40.833851 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:40.833993 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:40.834595 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:04:40.834717 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:04:40.834781 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:40.835023 (+ 242us) server_negotiation.cc:299] Negotiation successful
0812 01:04:40.835100 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399}
I20260812 01:04:40.835645 30044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 01:04:40.826977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42908 (local address 127.21.201.126:42797)
0812 01:04:40.827232 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0812 01:04:40.827242 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 01:04:40.827903 (+ 661us) server_negotiation.cc:407] Connection header received
0812 01:04:40.828457 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 01:04:40.828466 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 01:04:40.828690 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 01:04:40.828828 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 01:04:40.829666 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:40.830497 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:40.832843 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 01:04:40.833566 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 01:04:40.833692 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 01:04:40.834440 (+ 748us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 01:04:40.834890 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 01:04:40.835009 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 01:04:40.835217 (+ 208us) server_negotiation.cc:299] Negotiation successful
0812 01:04:40.835307 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 01:04:40.841034 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 29672
I20260812 01:04:40.842085 29778 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:04:40.958564 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 29672
I20260812 01:04:40.990670 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 29794
I20260812 01:04:40.992060 29900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:04:41.108348 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 29794
I20260812 01:04:41.137641 22309 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu pid 29922
I20260812 01:04:41.138862 30032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 01:04:41.255111 22309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJMOGj2/build/asan/bin/kudu with pid 29922
2026-08-12T01:04:41Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5035 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9907 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (402449 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST